Episode Details
Back to Episodes
Stanford CME295 Transformers and LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs
Description
For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education
To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/
Chapters:
00:00:00 Introduction
00:06:35 RL setup
00:09:32 Reward design
00:12:23 Policy gradient
00:17:12 Derivation of the gradient
00:23:50 REINFORCE
00:25:51 Importance sampling ratio
00:33:22 Challenges of policy gradients
00:40:34 Proximal Policy Optimization (PPO)
00:42:49 Value function and advantage
00:51:16 Policy ratio
00:53:03 Ratio clipping
00:57:45 KL Penalty
00:59:48 Connection with DPO
01:07:29 Group Relative Policy Optimization (GRPO)
01:09:42 Comparison of GRPO and PPO
01:18:25 Limitations of GRPO
01:22:41 DAPO, Dr. GRPO
01:25:17 On-Policy Distillation (OPD)
01:39:32 Self (OPSD) and Multi-Teacher (OPD) variants
Uploaded 19-10-26
Afshine Amidi is an Adjunct Professor at Stanford University.
Shervine Amidi is an Adjunct Professor at Stanford University.
Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X
Learn more about your ad choices. Visit megaphone.fm/adchoices