Abstract
Learning from human feedback plays an important role in aligning generativemodels, such as large language models (LLM). However, the effectiveness of thisapproach can be influenced by adversaries, who may intentionally providemisleading preferences to manipulate the output in an undesirable or harmfuldirection. To tackle this challenge, we study a specific model within thisproblem domain--contextual dueling bandits with adversarial feedback, where thetrue preference label can be flipped by an adversary. We propose an algorithmnamely robust contextual dueling bandit (\algo), which is based onuncertainty-weighted maximum likelihood estimation. Our algorithm achieves an$\tilde O(d\sqrt{T}+dC)$ regret bound, where $T$ is the number of rounds, $d$is the dimension of the context, and $ 0 \le C \le T$ is the total number ofadversarial feedback. We also prove a lower bound to show that our regret boundis nearly optimal, both in scenarios with and without ($C=0$) adversarialfeedback. Additionally, we conduct experiments to evaluate our proposedalgorithm against various types of adversarial feedback. Experimental resultsdemonstrate its superiority over the state-of-the-art dueling bandit algorithmsin the presence of adversarial feedback.