Courses / theory

Reinforcement Learning

advanced~18 hours16 lessonsEnvironment: pyodide

Prerequisites: python3, maths-for-robotics, ml-foundations-robotics

Start: When there is no teacher: rewards instead of labelsPractice questions

Modules

Module 1 — The RL problem
  1. 1When there is no teacher: rewards instead of labels
  2. 2MDPs: states, actions, rewards, discount — and the Markov assumption's teeth
  3. 3Reward design, and reward hacking
  4. 4Value functions and the Bellman equation
Module 2 — Tabular methods
  1. 1Q-learning from scratch
  2. 2Exploration vs exploitation: the dial with two failure ends
  3. 3Widget: watch Q-values propagate
  4. 4The curse returns: why tables die
Module 3 — From tables to networks
  1. 1Function approximation: the idea behind DQN
  2. 2Policy gradients: REINFORCE, derived and implemented
  3. 3Actor-critic: the modern workhorse
  4. 4Project: train a controller in NumPy (cart-pole / a 2D robot)
Module 4 — RL on real robots
  1. 1The sample-efficiency wall: why your robot cannot run 10 million episodes
  2. 2Sim-to-real: domain randomization and the reality gap
  3. 3Safety, resets, and the unglamorous truth about hardware RL
  4. 4Capstone: the reward-design workshop

Textbooks