10-403 Deep RL | Schedule

Date	Lecture	Readings	Logistics
M 01/13	Lecture #1 : Introduction to Reinforcement and Representation Learning [ slides ]	S & B Textbook, Ch1 (optional) Smith & Gasser. The Development of Embodied Cognition - Six Lessons from Babies (optional) Dan Wolpert't talk The real reason for brains
W 01/15	Lecture #2 : Multi-armed Bandits [ slides ]	Russo et al. A Tutorial on Thompson Sampling, Ch1-Ch4. Optional after Ch4 (optional) Aleksandrs Slivkins Introduction to Multi-Armed Bandits
F 01/17	Recitation #1: Neural Nets, PyTorch, OpenAI Gym, Bandits [ slides ]	G, B & C Textbook, Ch9, Ch10 Tensorflow tutorial notebook OpenAI Gym tutorial notebook PyTorch tutorial notebook The TensorFlow High Level (Keras) API
M 01/20	No Class, MLK Jr Day
W 01/22	Lecture #3 : Value-based Methods [ slides ]	S & B Textbook, Ch3, Ch4 The Path perspective on Value Learning (blogpost) DQN Neural Fitted Q-Iteration Policy Evaluation with Temporal Differences: A Survey and Comparison	HW1 out (tentative)
F 01/24	Recitation #2: Bandits, MDPs & HW1 [ slides ]
M 01/27	Lecture #4 : Value-based Methods (cont.) [ slides ]	S & B Textbook, Ch6, Ch7 7.1-7.3 S & B Textbook, Ch9.1-9.3, 9.6 and Ch10.1 DQN Neural Fitted Q-Iteration Policy Evaluation with Temporal Differences: A Survey and Comparison
W 01/29	Lecture #5 : Evolutionary Methods for Policy Search, Policy Gradients [ slides ]	S & B Textbook, Ch13 http://karpathy.github.io/2016/05/31/rl/ Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning (optional) Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial Policy gradients with function approximation Approximately Optimal Approximate RL On the theory of policy gradient methods
F 01/31	No Recitation
M 02/03	Lecture #6 : Actor-Critic Methods [ slides ]	Mnih et al. Asynchronous Methods for Deep Reinforcement Learning Off-Policy Actor Critic
W 02/05	Lecture #7 : Trust Region Constraints [ slides ]	(optional) Schulman et al. Trust Region Policy Optimization Schulman et al. Proximal Policy Optimization Algorithms (optional) Rajeswaran et al. Towards Generalization and Simplicity in Continuous Control https://arxiv.org/abs/2412.08442
F 02/07	Recitation #3: MCTS, TD Learning, Deep Q Learning, HW2 (DQN) [ slides ]
M 02/10	Lecture #8 : Maximum Entropy RL, Distributional RL, Deterministic Policy Gradients [ slides ]	Lillicrap et al. Continuous control with deep reinforcement learning Bellemare et al. A Distributional Perspective on Reinforcement Learning Haarnoja et al. Reinforcement Learning with Deep Energy Based Policies Haarnoja et al. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Fujimoto et al. Addressing Function Approximation Error in Actor-Critic Methods	HW1 due 11:59pm, HW2 out (tentative)
W 02/12	Lecture #9 : Imitation Learning I: Behavioral Cloning, Online imitation learning [ slides ]	Bojarski et al. End to End Learning for Self-Driving Cars Bagnell et al. An Invitation to Imitation Ross et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ho et al. Generative Adversarial Imitation Learning Fu et al. Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
F 02/14	Recitation #4: Quiz 1 Review [ slides ]
M 02/17	Lecture #10 : Imitation learning II: Multimodal policies, Multitask Policies [ slides ]	Luo Understanding Diffusion Models:A Unified Perspective Pearce et al. Imitating Human Behaviour with Diffusion Models https://arxiv.org/html/2410.24164v1 RT1 (and its follow-ups) Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
W 02/19	Lecture #11 : Visual Imitation [ slides ]	Peng et al. SFV:Reinforcement Learning of Physical Skills from Videos Baker et al. Video PreTraining (VPT):Learning to Act by Watching Unlabeled Online Videos
F 02/21	Quiz 1
M 02/24	Lecture #12 : Model Based RL I: Shooting / Collocation methods, Dyna [ slides ]	Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models Dyna https://arxiv.org/abs/1906.08253 Rybkin et al. Latent-space collocation
W 02/26	Lecture #13 : Model Based RL II: Online planning, MCTS, AlphaGo, AlphaZero [ slides ]	S & B Textbook, Ch8.11 (optional) Silver et al. Mastering the game of Go with deep neural networks and tree search Silver et al. Mastering the Game of Go without Human Knowledge (optional) David Silver et al. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm	HW2 due 11:59PM
F 02/28	Recitation #5: Solutions to Quiz 1 [ slides ]
M 03/03	Spring Break - No Classes
W 03/05	Spring Break - No Classes
F 03/07	Spring Break - No Classes
M 03/10	Lecture #14 : Model Based RL III: diffusion models and advanced topics [ slides ]	Janner et al. Planning with Diffusion for Flexible Behavior Synthesis Lee et al. Deep RL with a Latent Variable Model Hafner et al. Dreamer (and its subsequent versions)	HW3 out (tentative)
W 03/12	Lecture #15 : Summary so far - BUFFER [ slides ]
F 03/14	Recitation #6: Diffusion policies (cont.), HW3 Recitation [ slides ]
M 03/17	Lecture #16 : Exploration 1: Optimism in the face of uncertainty, count-based exploration, intrinsic motivation, RND, go explore [ slides ]	https://arxiv.org/abs/1606.01868 Ecoffet et al. Go-Explore: a New Approach for Hard-Exploration Problems Salimans et al. Learning Montezuma's Revenge from a Single Demonstration Exploration by random network distillation Count-Based Exploration with Neural Density Models Unifying Count-Based Exploration and Intrinsic Motivation
W 03/19	Lecture #17 : Exploration 2: curiosity, maximum entropy exploration, bootstrapped ensembles [ slides ]	Burda et al. Large-Scale Study of Curiosity-Driven Learning Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction Hazan et al. Provably Efficient Maximum Entropy Exploration Fu et al. EX2: Exploration with Exemplar Models for Deep Reinforcement Learning Osband et al. Deep Exploration via Randomized Value Functions
F 03/21	Recitation #7: MBRL in explicit and observable low-dimensional state spaces [ slides ]	Gonzalez et al. Learning to Simulate Complex Physics with Graph Networks Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
M 03/24	Lecture #18 : Offline RL 1: going beyond imitation, problem statement, challenges in doing offline RL, policy gradient methods / policy constraints [ slides ]	Kumar et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction Kumar et al. Reinforcement Learning from Static Datasets (Chapters 1 to 5) Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration Kumar and Levine Offline RL Tutorial, NeurIPS 2020	HW4 out (tentative)
W 03/26	Lecture #19 : Offline RL 2: conservative methods, model-based approaches, modern model-free algorithms [ slides ]	Kumar et al. Conservative Q-Learning for Offline Reinforcement Learning Kumar and Levine Offline RL Tutorial, NeurIPS 2020 Sobol Mark et al. Policy Agnostic RL Kostrikov et al. Offline RL with Implicit Q-Learning Yu et al. Conservative offline Model-Based Policy Optimization Kidambi et al. Model-Based Offline Reinforcement Learning
F 03/28	Quiz 2
M 03/31	Lecture #20 : RL for POMDPs / meta-RL: Belief-space RL, POMDP solutions different from MDP solutions, RL^2, information-gathering actions vs optimal actions, [ slides ]
W 04/02	Lecture #21 : sim2real history-conditioning + domain randomization robot locomotion, drone flying or robot acrobatics (all use PPO) [ slides ]	Kumar et al. Rapid Motor Adaptation for Legged Robots Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World Akkaya et al. Solving Rubik’s Cube with A Robot Hand https://arxiv.org/pdf/2406.10759
F 04/04	No Class, Spring Carnvial
M 04/07	Lecture #22 : Foundations models for RL [ slides ]	Liang et al.Code as Policies:Language Model Programs for Embodied Control https://arxiv.org/abs/2311.01455
W 04/09	Lecture #23 : RL for Training LLMs [ slides ]
F 04/11	Recitation #8: Homework 5 [ slides ]
M 04/14	Lecture #24 : RL for Training LLMs [ slides ]		HW5 out (tentative), HW4 due 11:59PM
W 04/16	Lecture #25 : [ slides ]
F 04/18	Recitation #9: No Recitation - Thanksgiving Break [ slides ]
M 04/21	Lecture #26 : [ slides ]
W 04/23	Lecture #27 : [ slides ]		HW5 due 11:59PM
F 04/25	Recitation #10: Quiz 3 Review [ slides ]