Faithful Preference Learning: Cognitively-Aligned Post-Training

Current post-training learns by interpreting human feedback as a reward to be maximized. However, human usually form preferences by comparing options over the long run, not by chasing a myopic, short score. Hence, treating feedback as a reward maximization does not match how people actually make judgements [1]. This project builds a novel technique of learning from human feedback beyond reward maximization that stays closer to real human judgment, and efficient even when human feedback is limited and costly to collect. Our approach has two parts:

The first is reading human feedback the way people actually produce it, which we call cognitive alignment. In our previous work PPL [2] and RePO [3], we model a preference as regret, meaning a human judgment of how much better a different choice would have been. This gives a more accurate picture of human decision-making, and in our experiments, it improved both alignment and efficiency at the same time. PPL began with robotic manipulation and RePO with LLMs, and this project extends the idea to agentic AI, where a system takes many steps and mixes human and automated feedback, tested on tool use benchmarks [4, 5].

The second part is keeping the learning provably correct when training runs at large scale asynchronously. Frontier post-training relies on asynchronous RL, which decouples rollout generation from policy updates to gain throughput across large clusters. This naturally leads to off-policy, because rollouts become stale and the offline data is heavily filtered and reused. In this regime, the estimators (e.g., importance ratios, divergence term) become unreliable and the policy gradient grows biased and high variance as staleness accumulates. Method such as GRPO [6] recover stability by normalization and hyperparameter tuning, yet no principled criterion tells us which of these corrections remain permissible [7]. This project characterizes how far the learned reward under ad-hoc off-policy corrections departs from the ground truth.