10-703 Deep RL | Schedule

Date	Lecture	Readings	Logistics
M 08/24	Lecture #1 : Welcome and Introduction to the Class [ slides \| slides 2 ]	S & B Textbook, Ch1 (optional) Smith & Gasser. The Development of Embodied Cognition - Six Lessons from Babies (optional) Dan Wolpert't talk The real reason for brains
W 08/26	Lecture #2 : Introduction to Reinforcement Learning [ slides ]	S & B Textbook, Ch1
F 08/28	Recitation #1: Neural Nets, PyTorch, Gymnasium [ slides ]	G, B & C Textbook, Ch9, Ch10 Gym tutorial notebook PyTorch tutorial notebook
M 08/31	Lecture #3 : Policy Gradient Methods [ slides ]	G, B & C Textbook, Ch13 http://karpathy.github.io/2016/05/31/rl/	HW1 out
W 09/02	Lecture #4 : Actor-Critic Methods (cont.) Evolutionary Methods for Policy Search [ slides ]	Mnih et al. Asynchronous Methods for Deep Reinforcement Learning Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning (optional) Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial
F 09/04	Recitation #2: MDPs, Policy Gradients, & HW1 [ slides ]
M 09/07	No Class, Labor Day
W 09/09	Lecture #5 : Value-based Methods [ slides ]	Mnih et al. Playing Atari with Deep Reinforcement Learning Martin Riedmiller Neural Fitted Q-Iteration Fu et al. Diagnosing Bottlenecks in Deep Q-learning Algorithms
F 09/11	Recitation #3: Value-Based Methods and Actor-Critic Methods & HW2 [ slides ]		HW2 Out HW1 Due
M 09/14	Lecture #6 : Value based methods (Cont.) [ slides ]	Deep Reinforcement Learning with Double Q-learning S & B Textbook, Ch8.11 Stop Regressing - Training Value Functions via Classification for Scalable Deep RL (optional) Policy Evaluation with Temporal Differences: A Survey and Comparison (optional) A Distributional Perspective on Reinforcement Learning
W 09/16	Lecture #7 : Advanced Policy Gradient Methods [ slides ]	Approximately Optimal Approximate RL Advantage-weighted regression Proximal Policy Optimization Algorithms
F 09/18	Recitation #4: Policy-Based Methods & HW3 [ slides ]
M 09/21	Lecture #8 : Advanced Policy Gradient Methods (Cont.) [ slides ]	Continuous control with deep reinforcement learning Addressing Function Approximation Error in Actor-Critic Methods Soft Actor-Critic Algorithms and Applications	HW3 Out HW2 Due
W 09/23	Lecture #9 : Off-Policy Actor-Critic and Max-Ent RL [ slides ]
F 09/25	Recitation #5: IL Diffusion Policies and HW4 [ slides ]
M 09/28	Lecture #10 : Imitation Learning [ slides ]	(optional) Chen et al. Learning by Cheating Bagnell. An Invitation to Imitation, Up To Page 10 Bojarski et al. End to End Learning for Self-Driving Cars Andrychowicz et al. Hindsight Experience Replay Ding et al. Goal-conditioned Imitation Learning
T 09/29	Project Description Out (tentative)
W 09/30	Lecture #11 : Imitation Learning and Diffusion [ slides ]		HW4 Out
F 10/02	Recitation #6: Midterm Review & HW4 [ slides ]		HW3 Due
M 10/05	Midterm (in class)
W 10/07	Lecture #12 : Reinforcement Learning & Generative Modeling [ slides ]
F 10/09	Recitation #7: IL Diffusion Policies [ slides ]		HW4 Due
M 10/12	Fall Break - No Classes
W 10/14	Fall Break - No Classes
F 10/16	Fall Break - No Classes
M 10/19	Lecture #13 : Offline RL [ slides ]	Kumar and Levine Offline RL Tutorial, NeurIPS 2020 Levine et al. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems Kumar et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction Fujimoto et al. A Minimalist Approach to Offline Reinforcement Learning
W 10/21	Lecture #14 : Model-based Methods [ slides ]	Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
F 10/23	Recitation #8: Midterm Solution Session & HW5 [ slides ]		Project Team Formation and Project Proposal Due (tentative)
M 10/26	Lecture #15 : Model-based Methods(Cont.) [ slides ]	(optional) Oh et al. Action-Conditional Video Prediction using Deep Networks in Atari Games (optional) Kaiser et al. Model-Based Reinforcement Learning for Atari Hansen et al. Temporal Difference Learning for Model Predictive Control	HW5 Out
W 10/28	Lecture #16 : Model-based Methods(Cont.), Learning and Tree-Search Planning [ slides ]	(optional) David Silver et al. Mastering the game of Go with deep neural networks and tree search David Silver et al. Mastering the game of Go without human knowledge David Silver et al. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
F 10/30	Recitation #9: TD-MPC & PETS [ slides ]
M 11/02	Lecture #17 : Model-based Methods for offline RL [ slides ]	Julian Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model Julian Schrittwieser et al. Online and Offline Reinforcement Learning by Planning with a Learned Model Kumar et al. Conservative Q-Learning for Offline Reinforcement Learning Rahul et al. MOReL - Model-Based Offline Reinforcement Learning
W 11/04	Lecture #18 : Guided Diffusion [ slides ]	Janner et al. Planning with Diffusion for Flexible Behavior Synthesis (optional) Yang et al. Diffusion-ES Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following
F 11/06	Recitation #10: Exploration [ slides ]
M 11/09	Lecture #19 : Exploration [ slides ]	Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction Burda et al. Exploration by Random Network Distillation
W 11/11	Lecture #20 : Sim2Real Learning [ slides ]	Akkaya et al. Solving Rubik’s Cube with A Robot Hand Kumar et al. Rapid Motor Adaptation for Legged Robots Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World	HW6 Out HW5 Due
F 11/13	Recitation #11: Sim2Real and HW6 [ slides ]
M 11/16	Lecture #21 : Sim2Real Learning (Cont.) R [ slides ]	Akkaya et al. Solving Rubik’s Cube with A Robot Hand Kumar et al. Rapid Motor Adaptation for Legged Robots Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
W 11/18	Lecture #22 : RL with Foundation Models [ slides ]	Ouyang et al. Training language models to follow instructions with human feedback Slides Direct Preference Optimization: A New RLHF Approach Xu et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study	Project Midway Report Due
F 11/20	Recitation #12: RL with Foundation Models [ slides ]
M 11/23	Lecture #23 : RL with Foundation Models (Cont.) [ slides ]	Ouyang et al. Training language models to follow instructions with human feedback Slides Direct Preference Optimization: A New RLHF Approach Xu et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study	HW7 Out HW6 Due
W 11/25	Thanksgiving Break - No Classes
F 11/27	Thanksgiving Break - No Recitation
M 11/30	Lecture #24 : Goal-Driven Reasoning for Multimodal AI Agents [ slides ]	Sarch et al. Grounded Reinforcement Learning for Visual Reasoning Sarch et al. VLM Agents Generate Their Own Memories
W 12/02	Lecture #25 : Projects [ slides ]
F 12/04	Lecture #26 : Projects [ slides ]		HW7 Due
M 12/07 - F 12/11	Project Final Report due (During Finals Week)