10-703 Deep RL | Schedule

Date	Lecture	Readings	Logistics
M 08/25	Lecture #1 : Welcome and Introduction to the Class [ slides \| slides 2 ]	S & B Textbook, Ch1 (optional) Smith & Gasser. The Development of Embodied Cognition - Six Lessons from Babies (optional) Dan Wolpert't talk The real reason for brains
W 08/27	Lecture #2 : Introduction to Reinforcement Learning [ slides ]	S & B Textbook, Ch1
F 08/29	Recitation #1: Neural Nets, PyTorch, Gymnasium [ slides \| notes ]	G, B & C Textbook, Ch9, Ch10 Gym tutorial notebook PyTorch tutorial notebook
M 09/01	No Class, Labor Day
W 09/03	Lecture #3 : Policy Gradient Methods [ slides ]	G, B & C Textbook, Ch13 http://karpathy.github.io/2016/05/31/rl/	HW1 out (tentative)
F 09/05	Recitation #2: MDPs, Policy Gradients, & HW1 [ slides ]
M 09/08	Lecture #4 : Actor-Critic Methods [ slides ]	Mnih et al. Asynchronous Methods for Deep Reinforcement Learning
W 09/10	Lecture #5 : Value-based Methods [ slides ]	S & B Textbook, Ch3, Ch4 S & B Textbook, Ch5, Ch6 (optional) The Path perspective on Value Learning (blogpost)
F 09/12	Recitation #3: Value-Based Methods and Actor-Critic Methods [ slides ]		HW1 Due, HW2 out (tentative)
M 09/15	Lecture #6 : Value based methods (Cont.) [ slides ]	DQN Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning S & B Textbook, Ch8.11 (optional) Neural Fitted Q-Iteration (optional) Policy Evaluation with Temporal Differences: A Survey and Comparison
W 09/17	Lecture #7 : Advanced Policy Gradient Methods [ slides ]	(optional) Schulman et al. Trust Region Policy Optimization (optional) Peng et al. Advantage-weighted regression (optional) Kakade and Langford Conservative policy iteration
F 09/19	Recitation #4: Value-Based Methods and Policy Gradient Methods [ slides ]		HW2 Due, HW3 Out (tentative)
M 09/22	Lecture #8 : Advanced Policy Gradient Methods (Cont.) [ slides ]	(optional) Schulman et al. Trust Region Policy Optimization (optional) Peng et al. Advantage-weighted regression (optional) Kakade and Langford Conservative policy iteration
W 09/24	Lecture #9 : Model-based Methods [ slides ]	Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning Janner et al. Planning with Diffusion for Flexible Behavior Synthesis Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models Janner et al. Model-based policy optimization Hafner et al. Dreamer
F 09/26	Recitation #5: TBD [ slides ]
M 09/29	Lecture #10 : Model-based Methods(Cont.) [ slides ]	Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning Janner et al. Planning with Diffusion for Flexible Behavior Synthesis Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models Janner et al. Model-based policy optimization Hafner et al. Dreamer	HW3 Due, HW4 Out (tentative)
T 09/30	Lecture #11 : [ slides ]		Project Description Out (tentative)
W 10/01	Lecture #12 : Imitation Learning [ slides ]	(optional) Chen et al. Learning by Cheating Bagnell. An Invitation to Imitation, Up To Page 10 Bojarski et al. End to End Learning for Self-Driving Cars
F 10/03	Recitation #6: Midterm [ slides ]
M 10/06	Lecture #13 : Imitation Learning(Cont.) [ slides ]	Peng et al. SFV Reinforcement Learning of Physical Skills from Videos Baker et al. Video PreTraining (VPT):Learning to Act by Watching Unlabeled Online Videos
W 10/08	Buffer
F 10/10	Recitation #7: TBD [ slides ]		HW4 Due (tentative), HW5 Out (tentative)
M 10/13	Fall Break - No Classes
W 10/15	Fall Break - No Classes
F 10/17	Fall Break - No Classes
M 10/20	Buffer
W 10/22	Lecture #14 : Offline RL [ slides ]	Kumar et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction Kumar et al. Reinforcement Learning from Static Datasets (Chapters 1 to 5) Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration Kumar and Levine Offline RL Tutorial, NeurIPS 2020
F 10/24	Recitation #8: TBD [ slides ]		Project Team Formation and Project Proposal Due (tentative)
M 10/27	Lecture #15 : Offline RL (Cont.) [ slides ]	Kumar et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction Kumar et al. Reinforcement Learning from Static Datasets (Chapters 1 to 5) Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration Kumar and Levine Offline RL Tutorial, NeurIPS 2020	HW5 Due, HW6 Out (tentative)
W 10/29	Lecture #16 : Exploration [ slides ]	Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction Burda et al. Exploration by Random Network Distillation
F 10/31	Recitation #9: TBD [ slides ]
M 11/03	Lecture #17 : Exploration (Cont.) [ slides ]	Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction Burda et al. Exploration by Random Network Distillation
W 11/05	Lecture #18 : Sim2Real Learning [ slides ]	Akkaya et al. Solving Rubik’s Cube with A Robot Hand Kumar et al. Rapid Motor Adaptation for Legged Robots Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
F 11/07	Recitation #10: TBD [ slides ]
M 11/10	Lecture #19 : Sim2Real Learning (Cont.) [ slides ]	Akkaya et al. Solving Rubik’s Cube with A Robot Hand Kumar et al. Rapid Motor Adaptation for Legged Robots Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World	HW6 Due, HW7 Out (tentative)
W 11/12	Lecture #20 : RL with Foundation Models [ slides ]	Ouyang et al. Training language models to follow instructions with human feedback Slides Direct Preference Optimization: A New RLHF Approach Xu et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
F 11/14	Recitation #11: TBD [ slides ]		Project Midway Report Due (tentative)
M 11/17	Lecture #21 : RL with Foundation Models (Cont.) [ slides ]	Ouyang et al. Training language models to follow instructions with human feedback Slides Direct Preference Optimization: A New RLHF Approach Xu et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
W 11/19	Lecture #22 : Generative Models for RL [ slides ]	Luo Understanding Diffusion Models:A Unified Perspective Pearce et al. Imitating Human Behaviour with Diffusion Models
F 11/21	Recitation #12: TBD [ slides ]
M 11/24	Recitation #13: Generative Models for RL (Cont.) [ slides ]		HW7 Due (tentative)
W 11/26	Thanksgiving Break - No Classes
F 11/28	Recitation #14: Thanksgiving Break - No Recitation [ slides ]
M 12/01	Lecture #23 : Guest Lecture [ slides ]
W 12/03	Lecture #24 : Course Review [ slides ]
F 12/05	Recitation #15: TBD [ slides ]
Finals Week	[ slides ]		Project Final Poster/Report due