Advanced
Reinforcement Learning
Agents that learn by trial and error, from bandits to policy gradients.
- MDPs, States, Actions & RewardsLearn · Free · 45 minA Markov Decision Process is five things: states, actions, transitions, rewards, a discount. Everything in RL — Q-learning, PPO, DPO, GRPO — optimizes over this shape. Learn it once, read the rest of reinforcement learning for free.Open
- Dynamic Programming — Policy Iteration & Value IterationBuild · 1 h 15 minDynamic programming is RL with cheating. You already know the transition and reward functions; you just iterate the Bellman equation until `V` or `π` stops moving. It is the benchmark every sampling-based method tries to approach.Needs Full Access
- Monte Carlo Methods — Learning from Complete EpisodesBuild · 1 h 15 minDynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the returns, average them. The simplest idea in RL — and the one that unlocks everything downstream.Needs Full Access
- Temporal Difference — Q-Learning & SARSABuild · 1 h 15 minMonte Carlo waits until the episode ends. TD updates after every step by bootstrapping the next value estimate. Q-learning is off-policy and optimistic; SARSA is on-policy and cautious. Both are one line of code. Both underpin every deep-RL method in this phase.Needs Full Access
- Deep Q-Networks (DQN)Build · 1 h 15 min2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL era. DQN is Q-learning plus three tricks that make function approximation stable.Needs Full Access
- Policy Gradient — REINFORCE from ScratchBuild · 1 h 15 minStop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill. Williams (1992) wrote it in one theorem. It is why PPO, GRPO, and every LLM RL loop exist.Needs Full Access
- Actor-Critic — A2C and A3CBuild · 1 h 15 minREINFORCE is noisy. Add a critic that learns `V̂(s)`, subtract it from the return, and you get an advantage that has the same expectation but far lower variance. That is actor-critic. A2C runs it synchronously; A3C runs it across threads. Both are the mental model for every modern deep-RL method.Needs Full Access
- Proximal Policy Optimization (PPO)Build · 1 h 15 minA2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do 10+ epochs on the same data without the policy exploding. Schulman et al. (2017). Still the default policy-gradient algorithm in 2026.Needs Full Access
- Reward Modeling & RLHFBuild · 45 minHumans cannot write a reward function for "good assistant response," but they can compare two responses and pick the better one. Fit a reward model to those comparisons, then RL the language model against it. Christiano 2017. InstructGPT 2022. The recipe that turned GPT-3 into ChatGPT. In 2026 it is mostly being replaced by DPO — but the mental model stays.Needs Full Access
- Multi-Agent RLBuild · 45 minSingle-agent RL assumes the environment is stationary. Put two learning agents in the same world and that assumption breaks: each agent is part of the other's environment, and both are changing. Multi-agent RL is the set of tricks to make learning converge when the Markov assumption no longer holds.Needs Full Access
- Sim-to-Real TransferLearn · 45 minA policy trained in a simulator that fails on hardware is a policy that memorized the simulator. Domain randomization, domain adaptation, and system identification are the three tools to make learned controllers cross the reality gap.Needs Full Access
- RL for Games — AlphaZero, MuZero, and the LLM-Reasoning EraBuild · 2 h1992: TD-Gammon beat human champions at backgammon with pure TD. 2016: AlphaGo beat Lee Sedol. 2017: AlphaZero dominated chess, shogi, and Go from scratch. 2024: DeepSeek-R1 proved the same recipe, with GRPO replacing PPO, works on reasoning. Games are the benchmark that drives every breakthrough in this phase.Needs Full Access