Reinforcement Learning Journal, vol. TBD, 2025, pp. TBD.
Presented at the Reinforcement Learning Conference (RLC), Edmonton, Alberta, Canada, August 5–9, 2025.
Large Language Models (LLMs) have made significant strides in generating human-like responses, largely due to preference alignment techniques. However, these methods often assume unbiased human feedback, which is rarely the case in real-world scenarios. This paper introduces Content-Aware Noise-Resilient Preference Optimization (CNRPO), a novel framework that addresses multiple sources of content-dependent noise in preference learning. CNRPO employs a multi-objective optimization approach to separate true preferences from content-aware noises, effectively mitigating their impact. We leverage backdoor attack mechanisms to efficiently learn and control various noise sources within a single model. Theoretical analysis and extensive experiments on different synthetic noisy datasets demonstrate that CNRPO significantly improves alignment with primary human preferences while controlling for secondary noises and biases, such as response length and harmfulness.
Amirabbas Afzali, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi, and Sanjay Lall. "One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware, Multi-Source Noise." Reinforcement Learning Journal, vol. TBD, 2025, pp. TBD.
BibTeX:@article{afzali2025goal,
title={One Goal, Many Challenges: {R}obust Preference Optimization Amid Content-Aware, Multi-Source Noise},
author={Afzali, Amirabbas and Afsharrad, Amirhossein and Mousavi, Seyed Shahabeddin and Lall, Sanjay},
journal={Reinforcement Learning Journal},
year={2025}
}