Episode Details

Daniel wants a model that generates episode titles and descriptions, learns from side-by-side button presses, and improves each checkpoint. Herman walks through what that pipeline actually is: an iterative preference optimization loop built on DPO (Direct Preference Optimization), with an implicit reward derived from a frozen reference model rather than a separate reward model or RL rollout. Along the way: what one saved record must contain, why ties deserve their own objective (DPO-RK, DPO-D), and why a single transcript's four candidates are still just one opinion. Key sources: • Rafailov et al., Direct Preference Optimization . — https://arxiv.org/abs/2305.18290 • Chen et al., On Extending DPO to Accommodate Ties . — https://arxiv.org/abs/2409.17431 • author's NeurIPS 2025 summary . — https://www.jinghong-chen.net/my-on-extending-direct-preference-optimization-to-accommodate-ties-neurips-2025/ Episode 6242 — myweirdprompts.com/n/6242 (permanent short link: myweirdprompts.com/171958)