10-403 Deep RL | Schedule

Date	Lecture	Readings	Logistics
T 02/02	Lecture #1 : Introduction [ slides \| video ]	S & B Textbook, Ch1 Smith & Gasser. The Development of Embodied Cognition - Six Lessons from Babies Dan Wolpert't talk The real reason for brains
Th 02/04	Lecture #2 : Evolutionary methods for policy search [ slides \| video ]	Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial, optional
F 02/05	Recitation #1: Neural Nets, TensorFlow & Keras, OpenAI Gym [ slides \| slides 2 \| video ]	G, B & C Textbook, Ch9, Ch10 Tensorflow tutorial notebook OpenAI Gym tutorial notebook
T 02/09	Lecture #3 : Evolutionary methods for policy search (cont.) [ slides \| video ]	Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial, optional
Th 02/11	Lecture #4 : Multi-armed bandits [ slides \| video ]	S & B Textbook, Ch2 2.1-2.7 Russo et al. A Tutorial on Thompson Sampling, Ch1-Ch4. Optional after Ch4	HW1 out
F 02/12	Recitation #2: Gaussian Processes [ slides \| video ]
T 02/16	Lecture #5 : Bayesian Optimization with Gaussian Processes [ slides \| video ]	Cully et al. Robots that can adapt like animals, discussed in lecture Rasmussen & Williams Gaussian Processes for Machine Learning Ch1, Ch2 2.1-2.3 Brochu et. al A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning Sections 1, 2.1, 2.2
Th 02/18	Lecture #6 : Bayesian Optimization with Gaussian Processes (cont.) [ slides \| video ]	Cully et al. Robots that can adapt like animals, discussed in lecture Rasmussen & Williams Gaussian Processes for Machine Learning Ch1, Ch2 2.1-2.3 Brochu et. al A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning Sections 1, 2.1, 2.2
F 02/19	Recitation #3: HW1 [ slides \| video ]
T 02/23	Break day - No classes
Th 02/25	Lecture #7 : Imitation learning with behavior cloning [ slides \| video ]	Chen et al. Learning by Cheating Bojarski et al. End to End Learning for Self-Driving Cars, discussed in lecture Bansal et al. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, optional Bagnell. An Invitation to Imitation, up to Page 10 Ross et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, optional
F 02/26	Recitation #4: REC: Review of MDP, Value Functions, Bellman Equation [ slides \| video ]		HW1 due 02/26 11:59pm
T 03/02	Lecture #8 : Markov Decision Processes, Value Iteration, Policy Iteration [ slides \| video ]	S & B Textbook, Ch3, Ch4 The Path perspective on Value Learning (blogpost)
Th 03/04	Lecture #9 : Monte Carlo Learning and Temporal Difference Learning [ slides \| video ]	S & B Textbook, Ch5, Ch6
F 03/05	Recitation #5: A comparative overview of DP vs. MC vs. TD [ slides \| video ]		HW2 out
T 03/09	Lecture #10 : Monte Carlo Learning and Temporal Difference Learning (cont.) [ slides \| video ]	S & B Textbook, Ch5, Ch6
Th 03/11	Lecture #11 : Function approximation in prediction and control [ slides \| video ]	S & B Textbook, Ch6, Ch7 7.1-7.3 Mnih et al. Playing Atari with Deep Reinforcement Learning
F 03/12	Quiz 1 (online)
T 03/16	Lecture #12 : Monte Carlo Tree search [ slides \| video ]	S & B Textbook, Ch 8.11 Pritzel et al. Neural Episodic Control, discussed in lecture Guo et al. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
Th 03/18	Lecture #13 : REINFORCE, Actor-Critic methods [ slides \| video ]	S & B Textbook, Ch13 Deep Reinforcement Learning: Pong from Pixels (blogpost) Mnih et al. Asynchronous Methods for Deep Reinforcement Learning
F 03/19	Mid-semester break - No classes HW2 due 03/19 11:59pm HW3 out
T 03/23	Lecture #14 : REINFORCE, Actor-Critic methods (cont.) [ slides \| video ]	Mnih et al. Asynchronous Methods for Deep Reinforcement Learning Lillicrap et al. Continuous control with deep reinforcement learning
Th 03/25	Lecture #15 : REINFORCE, Actor-Critic methods (cont.), Natural PG [ slides \| slides 2 \| video ]	Natural Gradient Descent (blogpost) Schulman et al. Proximal Policy Optimization Algorithms Schulman et al. Trust Region Policy Optimization
F 03/26	Recitation #6: Quiz 1 Review [ slides ]
T 03/30	Lecture #16 : Natural PG (cont.), Deterministic PG, Re-parametrized PG [ slides \| slides 2 \| video ]	Schulman et al. Proximal Policy Optimization Algorithms Schulman et al. Trust Region Policy Optimization Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration Doersch Tutorial on Variational Autoencoders
Th 04/01	Lecture #17 : MCTS with prior knowledge [ slides \| video ]	Silver et al. Mastering the game of Go with deep neural networks and tree search Silver et al. Mastering the game of Go without human knowledge
F 04/02	Recitation #7: Quiz 2 Review [ slides \| video ]		HW3 due 04/02 11:59PM
T 04/06	Lecture #18 : Model-based RL [ slides \| video ]	Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models Janner et al. When to Trust Your Model: Model-Based Policy Optimization Kurutach et al. Model-Ensemble Trust-Region Policy Optimization Oh et al. Action-Conditional Video Prediction using Deep Networks in Atari Games Kaiser et al. Action-Conditional Video Prediction using Deep Networks in Atari Games
Th 04/08	Lecture #19 : Model-based RL II: graph neural networks, iLQR [ slides \| video ]	Battaglia et al. Interaction Networks for Learning about Objects, Relations and Physics Gonzalez et al. Learning to Simulate Complex Physics with Graph Networks Gonzalez et al. Graph Networks as Learnable Physics Engines for Inference and Control Qi et al. Learning Long-term Visual Dynamics with Region Proposal Interaction Networks Tutorial - Representation Learning on Networks
F 04/09	Quiz 2 (online) HW4 out
T 04/13	Lecture #20 : MBRL with latent dynamic models [ slides \| video ]	Yan et al. Learning Predictive Representations for Deformable Objects using Contrastive Estimation Agrawal et al. Learning to Poke by Poking: Experiential Learning of Intuitive Physics Hafner et al. Dream to Control:Learning Behaviors by Latent Imagination
Th 04/15	No classes
F 04/16	Spring Carnival - No classes
T 04/20	Lecture #21 : Exploration using curiosity and episodic memory [ slides \| video ]	Osband et al. Deep Exploration via Bootstrapped DQN Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction Burda et al. Large-Scale Study of Curiosity-Driven Learning Savinov et al. Episodic Curiosity through Reachability Eccofet et al. First return, then explore Salimans et al. Learning Montezuma's Revenge from a Single Demonstration
Th 04/22	Lecture #22 : Learning from demonstrations and task rewards, off-policy RL, adversarial imitation learning [ slides \| video ]	Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration Ho et al. Generative Adversarial Imitation Learning Zhu et al. Reinforcement and Imitation Learning for Diverse Visuomotor Skills
F 04/23	Recitation #8: Distributional RL [ slides \| video ]		HW4 due 04/23 11:59pm
T 04/27	Lecture #23 : Visual Imitation Learning [ slides \| video ]	Aytar et al. Playing hard exploration games by watching YouTube Peng et al. SFV : Reinforcement Learning of Physical Skills from Videos Mandlekar et al. IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
Th 04/29	Lecture #24 : Visual Imitation Learning (cont.), Sim2Real transfer [ slides \| video ]	Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World Muller et al. Driving Policy Transfer via Modularity and Abstraction Akkaya et al. Solving Rubik’s Cube with A Robot Hand Zeng et al. TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
T 05/04	Lecture #25 : State representation learning for RL [ slides \| video ]	He et al. Momentum Contrast for Unsupervised Visual Representation Learning Srinivas et al. CURL: Contrastive Unsupervised Representations for Reinforcement Learning
Th 05/06	Lecture #26 : RL and generalization: A closer look to state representations for generalization in model free and model based RL [ slides \| video ]	Florence et al. Self-Supervised Correspondence in Visuomotor Policy Learning Tung et al. Visually-Grounded Library of Behaviours for Transfering Manipulation across Objects and Views Zambaldi et al. Relational Deep Reinforcement Learning Google AI Blog: Using Selective Attention in Reinforcement Learning Agents Ding et al. Mutual Information Maximization for Robust Plannable Representations
F 05/07	Recitation #9: Quiz 3 Review [ slides \| video ]
Mo 05/10	Quiz 3 8:30AM - 11:30AM

T 02/02

Lecture #1 :
Introduction
[ slides | video ]

S & B Textbook, Ch1
Smith & Gasser. The Development of Embodied Cognition - Six Lessons from Babies
Dan Wolpert't talk The real reason for brains

Th 02/04

Lecture #2 :
Evolutionary methods for policy search
[ slides | video ]

Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial, optional

F 02/05

Recitation #1:
Neural Nets, TensorFlow & Keras, OpenAI Gym
[ slides | slides 2 | video ]

G, B & C Textbook, Ch9, Ch10
Tensorflow tutorial notebook
OpenAI Gym tutorial notebook

T 02/09

Lecture #3 :
Evolutionary methods for policy search (cont.)
[ slides | video ]

Salimans et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Nikolaus Hansen. The CMA Evolution Strategy - A Tutorial, optional

Th 02/11

Lecture #4 :
Multi-armed bandits
[ slides | video ]

S & B Textbook, Ch2 2.1-2.7
Russo et al. A Tutorial on Thompson Sampling, Ch1-Ch4. Optional after Ch4

HW1 out

F 02/12

Recitation #2:
Gaussian Processes
[ slides | video ]

T 02/16

Lecture #5 :
Bayesian Optimization with Gaussian Processes
[ slides | video ]

Cully et al. Robots that can adapt like animals, discussed in lecture
Rasmussen & Williams Gaussian Processes for Machine Learning Ch1, Ch2 2.1-2.3
Brochu et. al A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning Sections 1, 2.1, 2.2

Th 02/18

Lecture #6 :
Bayesian Optimization with Gaussian Processes (cont.)
[ slides | video ]

Cully et al. Robots that can adapt like animals, discussed in lecture
Rasmussen & Williams Gaussian Processes for Machine Learning Ch1, Ch2 2.1-2.3
Brochu et. al A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning Sections 1, 2.1, 2.2

F 02/19

Recitation #3:
HW1
[ slides | video ]

T 02/23

Break day - No classes

Th 02/25

Lecture #7 :
Imitation learning with behavior cloning
[ slides | video ]

Chen et al. Learning by Cheating
Bojarski et al. End to End Learning for Self-Driving Cars, discussed in lecture
Bansal et al. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst, optional
Bagnell. An Invitation to Imitation, up to Page 10
Ross et al. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, optional

F 02/26

Recitation #4:
REC: Review of MDP, Value Functions, Bellman Equation
[ slides | video ]

HW1 due 02/26 11:59pm

T 03/02

Lecture #8 :
Markov Decision Processes, Value Iteration, Policy Iteration
[ slides | video ]

S & B Textbook, Ch3, Ch4
The Path perspective on Value Learning (blogpost)

Th 03/04

Lecture #9 :
Monte Carlo Learning and Temporal Difference Learning
[ slides | video ]

S & B Textbook, Ch5, Ch6

F 03/05

Recitation #5:
A comparative overview of DP vs. MC vs. TD
[ slides | video ]

HW2 out

T 03/09

Lecture #10 :
Monte Carlo Learning and Temporal Difference Learning (cont.)
[ slides | video ]

S & B Textbook, Ch5, Ch6

Th 03/11

Lecture #11 :
Function approximation in prediction and control
[ slides | video ]

S & B Textbook, Ch6, Ch7 7.1-7.3
Mnih et al. Playing Atari with Deep Reinforcement Learning

F 03/12

Quiz 1 (online)

T 03/16

Lecture #12 :
Monte Carlo Tree search
[ slides | video ]

S & B Textbook, Ch 8.11
Pritzel et al. Neural Episodic Control, discussed in lecture
Guo et al. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning

Th 03/18

Lecture #13 :
REINFORCE, Actor-Critic methods
[ slides | video ]

S & B Textbook, Ch13
Deep Reinforcement Learning: Pong from Pixels (blogpost)
Mnih et al. Asynchronous Methods for Deep Reinforcement Learning

F 03/19

Mid-semester break - No classes

HW2 due 03/19 11:59pm
HW3 out

T 03/23

Lecture #14 :
REINFORCE, Actor-Critic methods (cont.)
[ slides | video ]

Mnih et al. Asynchronous Methods for Deep Reinforcement Learning
Lillicrap et al. Continuous control with deep reinforcement learning

Th 03/25

Lecture #15 :
REINFORCE, Actor-Critic methods (cont.), Natural PG
[ slides | slides 2 | video ]

Natural Gradient Descent (blogpost)
Schulman et al. Proximal Policy Optimization Algorithms
Schulman et al. Trust Region Policy Optimization

F 03/26

Recitation #6:
Quiz 1 Review
[ slides ]

T 03/30

Lecture #16 :
Natural PG (cont.), Deterministic PG, Re-parametrized PG
[ slides | slides 2 | video ]

Schulman et al. Proximal Policy Optimization Algorithms
Schulman et al. Trust Region Policy Optimization
Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration
Doersch Tutorial on Variational Autoencoders

Th 04/01

Lecture #17 :
MCTS with prior knowledge
[ slides | video ]

Silver et al. Mastering the game of Go with deep neural networks and tree search
Silver et al. Mastering the game of Go without human knowledge

F 04/02

Recitation #7:
Quiz 2 Review
[ slides | video ]

HW3 due 04/02 11:59PM

T 04/06

Lecture #18 :
Model-based RL
[ slides | video ]

Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Janner et al. When to Trust Your Model: Model-Based Policy Optimization
Kurutach et al. Model-Ensemble Trust-Region Policy Optimization
Oh et al. Action-Conditional Video Prediction using Deep Networks in Atari Games
Kaiser et al. Action-Conditional Video Prediction using Deep Networks in Atari Games

Th 04/08

Lecture #19 :
Model-based RL II: graph neural networks, iLQR
[ slides | video ]

Battaglia et al. Interaction Networks for Learning about Objects, Relations and Physics
Gonzalez et al. Learning to Simulate Complex Physics with Graph Networks
Gonzalez et al. Graph Networks as Learnable Physics Engines for Inference and Control
Qi et al. Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
Tutorial - Representation Learning on Networks

F 04/09

Quiz 2 (online)

HW4 out

T 04/13

Lecture #20 :
MBRL with latent dynamic models
[ slides | video ]

Yan et al. Learning Predictive Representations for Deformable Objects using Contrastive Estimation
Agrawal et al. Learning to Poke by Poking: Experiential Learning of Intuitive Physics
Hafner et al. Dream to Control:Learning Behaviors by Latent Imagination

Th 04/15

No classes

F 04/16

Spring Carnival - No classes

T 04/20

Lecture #21 :
Exploration using curiosity and episodic memory
[ slides | video ]

Osband et al. Deep Exploration via Bootstrapped DQN
Pathak et al. Curiosity-driven Exploration by Self-supervised Prediction
Burda et al. Large-Scale Study of Curiosity-Driven Learning
Savinov et al. Episodic Curiosity through Reachability
Eccofet et al. First return, then explore
Salimans et al. Learning Montezuma's Revenge from a Single Demonstration

Th 04/22

Lecture #22 :
Learning from demonstrations and task rewards, off-policy RL, adversarial imitation learning
[ slides | video ]

Fujimoto et al. Off-Policy Deep Reinforcement Learning without Exploration
Ho et al. Generative Adversarial Imitation Learning
Zhu et al. Reinforcement and Imitation Learning for Diverse Visuomotor Skills

F 04/23

Recitation #8:
Distributional RL
[ slides | video ]

HW4 due 04/23 11:59pm

T 04/27

Lecture #23 :
Visual Imitation Learning
[ slides | video ]

Aytar et al. Playing hard exploration games by watching YouTube
Peng et al. SFV : Reinforcement Learning of Physical Skills from Videos
Mandlekar et al. IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Th 04/29

Lecture #24 :
Visual Imitation Learning (cont.), Sim2Real transfer
[ slides | video ]

Tobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
Muller et al. Driving Policy Transfer via Modularity and Abstraction
Akkaya et al. Solving Rubik’s Cube with A Robot Hand
Zeng et al. TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

T 05/04

Lecture #25 :
State representation learning for RL
[ slides | video ]

He et al. Momentum Contrast for Unsupervised Visual Representation Learning
Srinivas et al. CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Th 05/06

Lecture #26 :
RL and generalization: A closer look to state representations for generalization in model free and model based RL
[ slides | video ]

Florence et al. Self-Supervised Correspondence in Visuomotor Policy Learning
Tung et al. Visually-Grounded Library of Behaviours for Transfering Manipulation across Objects and Views
Zambaldi et al. Relational Deep Reinforcement Learning
Google AI Blog: Using Selective Attention in Reinforcement Learning Agents
Ding et al. Mutual Information Maximization for Robust Plannable Representations

F 05/07

Recitation #9:
Quiz 3 Review
[ slides | video ]

Mo 05/10

Quiz 3

8:30AM - 11:30AM