On the Role of Preference Variance in Preference Optimization

This academic paper investigates the concept of Preference Variance (PVar) as a metric for improving the efficiency of Direct Preference Optimization (DPO), a method for aligning large language models (LLMs) with human feedback. The authors establish a theoretical foundation demonstrating that the magnitude of the DPO training gradient is bounded by the PVar of a given prompt, meaning prompts with low PVar contribute minimally to learning. Experimentally, the paper validates that training LLMs using subsets of data identified as having high PVar leads to faster convergence and superior performance compared to using randomly selected data or the entire dataset. Ultimately, the research suggests that strategically selecting high-PVar prompts can drastically reduce the cost of human annotation while maintaining or even improving the final quality of LLM alignment.