Learning Resources

Readings

Chapters from these two texts, as well as relevant academic papers, will be referenced alongside each lecture, under the Lectures section.

The papers and chapters linked to each lecture are required unless noted as optional. They are instrumental to help you understand the concepts and algorithms taught in class.

If you want additional resources which contain parts of the course material, we recommend the following:

Online courses

Recitations

Recitations are usually given by TAs and their goal is to cover background material needed to understand the course, as well as to provide help for homeworks. You can control the material of the recitations every week by posting on Piazza the topics you wish to be discussed on friday’s recitation. Take advantage of this privilege to better tailor the course to your needs.

Communication

Piazza is intended for all announcements, general questions about the course, clarifications about assignments, student questions to each other, discussions about material, and so on.

We strongly encourage all students to participate in discussion, ask, and answer questions in class as well as through Piazza!

We strongly encourage all students to form reading groups for joint study of the papers and the materials, to make the semester more fun and more productive.

Grading

The class requirements include 5 homework assignments (55% of the grade) and 3 quizzes (45% of the grade). By the end of this class you should have a good understanding of the basic methodologies in deep reinforcement learning, and be able to use them to solve real problems of modest complexity.

The entirety of your course grade will be based on the performance in the homework assignments and the quizzes.

Individual homeworks may NOT necessarily be worth the same percentage. Some homework assignments may be worth more points than others, therefore weighing more towards the total.

Homework Assignments

There will be 5 homework assignments over the course of the semester. For each, you may work in groups of two or three students. Only one person should submit the writeup and code on Gradescope. Make sure you mark your partner/s as a collaborator on Gradescope and that both names are listed in the writeup.

These assignments may contain material that has been covered by published papers and webpages. We expect students to solve the problems themselves rather than search for answers.

Students are required to typeset written homework solutions using \(\LaTeX\) and the provided template, and submit as PDF. All code, including auxiliary scripts used for testing should be submitted with a README file to explain/document them.

Quizzes

All quizzes will be in-person and taken in the classroom on Gradescope using your personal laptop. We will use up to the full recitation time for quizzes 1 and 2. Please, make sure your device is sufficiently charged and bring your charger as backup. We cannot guarantee one power outlet per person. Students in the remote section will will take the exam concurrently during the class period. As an FYI, the last quiz is 3 hours and the date and time is set by the registrar.

Quiz 3 will consist of two parts. (i) A question part that is analogous to the first two quizzes which will test your understanding of all topics covered in this class including the assigned readings. There will be an emphasis on question that target content after Quiz 2; (ii) A paper reading part where we select a new (i.e., one not covered in class) RL paper for you to read and you will have to answer a set of questions that test your ability to understand the paper.

The quizzes are open book, though no google search (or any online search) is allowed. You may can only access downloaded notes and papers. The quizzes are not cumulative and will only test a subset of the material, but they can have questions that require you to know older material.

Each quiz will count equally toward the overall quiz grade.

Remote Student Engagement

The remote-only (REO) option for this course is for students who cannot be on campus due to a government restriction on visas or travel or a documented health issue. To ensure that we are supporting these remote students:

  • Students in the remote section will be included in the course discussion forum, Piazza, to ask and answer questions.
  • Remote students will receive feedback on their assignments via marked rubric items in Gradescope, as all students do.
  • We encourage students to request office hours by appointment as needed. Please contact the course staff if you need more support in office hours.
  • Lectures and recitations will be recorded and made available to students. There may be a delay in posting these recordings while they are processed.
  • Exams will take place at the same time and place as the in-person section. We will require students in the remote section to attend the in-person exams if they are able. Accommodations for students to take the exam remotely at the same day and time as the in-person section will be considered on a case-by-case basis.

AWS Resources

For those of you who need GPU resources for future homeworks, please read through this section carefully.

  • If you are not officially registered for this class, you are not allowed to request resources. We will be checking before we submit requests, so please do not request access to them.
  • We will be offering AWS resources. All students should join AWS educate using this link: https://aws.amazon.com/education/awseducate/ using their @andrew.cmu.edu email address. If you do not use your andrew email address, your resources may be denied. You should do this as soon as possible, as it can take time to set up your accounts.
  • AWS NOTE: You need to back this account with your own credit/debit card and we will give out allocation codes of $50, this is important as should you go over this $50 it will start charging to your card, please be sure to keep an eye on your funds and not forget to terminate instances. The university holds no responsibility in paying for additional usage.
  • We will ask you to complete an Allocation Form in order to apply for your resources. This will be made available later in the semester. Note: HW1 does not require AWS resources.

Collaboration Policy

(Adapted from Roni Rosenfeld’s 10-601 Spring 2016 Course Policies.)

  • The purpose of student collaboration is to facilitate learning, not to circumvent it. Studying the material in groups is strongly encouraged. It is also allowed to seek help from other students in understanding the material needed to solve a particular homework problem, provided no written notes (including code) are shared, or are taken at that time, and provided learning is facilitated, not circumvented. The actual solution must be done by each student alone.

  • The presence or absence of any form of help or collaboration, whether given or received, must be explicitly stated and disclosed in full by all involved. Specifically, each assignment solution must include answering the following questions:

  1. Did you receive any help whatsoever from anyone in solving this assignment? Yes / No.
    • If you answered ‘yes’, give full details: ____
    • (e.g. “Jane Doe explained to me what is asked in Question 3.4”)
  2. Did you give any help whatsoever to anyone in solving this assignment? Yes / No.
    • If you answered ‘yes’, give full details: _____
    • (e.g. “I pointed Joe Smith to section 2.3 since he didn’t know how to proceed with Question 2”)
  3. Did you find or come across code that implements any part of this assignment ? Yes / No. (See below policy on “found code”)
    • If you answered ‘yes’, give full details: _____
    • (book & page, URL & location within the page, etc.).

  • If you gave help after turning in your own assignment and/or after answering the questions above, you must update your answers before the assignment’s deadline, if necessary by emailing the course staff.

  • Collaboration without full disclosure will be handled severely, in compliance with CMU’s Policy on Cheating and Plagiarism.

Academic Integrity (Read this carefully!)

Previously Used Assignments

Some of the homework assignments used in this class may have been used in prior versions of this class, or in classes at other institutions, or elsewhere. Solutions to them may be, or may have been, available online, or from other people or sources.

It is explicitly forbidden to use any such sources, or to consult people who have solved these problems before. It is explicitly forbidden to search for these problems or their solutions on the internet. You must solve the homework assignments completely on your own. We will be actively monitoring your compliance.

Collaboration with other students who are currently taking the class is allowed, but only under the conditions stated above.

Policy Regarding “Found Code”

You are encouraged to read books and other instructional materials, both online and offline, to help you understand the concepts and algorithms taught in class. These materials may contain example code or pseudo code, which may help you better understand an algorithm or an implementation detail.

However, when you implement your own solution to an assignment, you must put all materials aside, and write your code completely on your own, starting from scratch. Specifically, you may not use any code you found or came across.

If you find or come across code that implements any part of your assignment, you must disclose this fact in your collaboration statement.

Duty to Protect One’s Work

Students are responsible for pro-actively protecting their work from copying and misuse by other students. If a student’s work is copied by another student, the original author is also considered to be at fault and in gross violation of the course policies. It does not matter whether the author allowed the work to be copied or was merely negligent in preventing it from being copied. When overlapping work is submitted by different students, both students will be punished.

To protect future students, do not post your solutions publicly, neither during the course nor afterwards.

Penalties for Violations of Course Policies

All violations (even the first violation) of the academic integrity or collaboration policy will always be reported to the university authorities (your Department Head, Associate Dean, Dean of Student Affairs, etc.) as an official Academic Integrity Violation and will carry severe penalties.

  1. The penalty for the first violation is a one-and-a-half letter grade reduction. For example, if your final letter grade for the course was to be an A-, it would become a C+.

  2. The penalty for the second violation is failure in the course, and can even lead to dismissal from the university.

Late Policy

Each student has a total of 8 grace days that may be applied to the homework assignments. No more than 3 grace days may be used on any single assignment.

Any assignment submitted more than 3 days past the deadline (or the date the student no longer has late day credit) will get zero credit.

Grace days will be subtracted from all students in the homework team. E.g. an assignment submitted 1 day late by a team of 2 people will result in both team members losing 1 grace day from their total allotment.

Regrade Policy

If you feel that we have made a mistake in grading your homework, please request a regrade on Gradescope and we will consider your request.

Please note that regrading of a homework may cause your grade to go either up or down.

Extensions

In general, we do not grant extensions on assignments. There are several exceptions:

  • Medical Emergencies: If you are sick and unable to complete an assignment or attend class, please go to University Health Services. For minor illnesses, we expect grace days or our late penalties to provide sufficient accommodation. For medical emergencies (e.g. prolonged hospitalization), students may request an extension afterwards and should include a note from University Health Services.

  • Family/Personal Emergencies: If you have a family emergency (e.g. death in the family) or a personal emergency (e.g. mental health crisis), please contact your academic adviser or Counseling and Psychological Services (CaPS). In addition to offering support, they will reach out to the instructors for all your courses on your behalf to request an extension.

  • University-Approved Absences: If you are attending an out-of-town university approved event (e.g. multi-day athletic/academic trip organized by the university), you may request an extension for the duration of the trip. You must provide confirmation of your attendance, usually from a faculty or staff organizer of the event.

For any of the above situations, you may request an extension by emailing Joshmin. The email should be sent as soon as you are aware of the conflict and at least 5 days prior to the deadline. In the case of an emergency, no notice is needed.

Pass/Fail Policy

We allow you take the course as Pass/Fail. Instructor permission is not required. You must complete all aspects of the course (all homeworks and quizzes) if you take the course as Pass/Fail.

What grade is the cutoff for Pass will depend on your program. Be sure to check with your program / department as to whether you can count a Pass/Fail course towards your degree requirements, notify us that you want to take the course Pass/Fail, and notify us of the Pass threshold your department uses (i.e., does it correspond to a grade of A, B, C, or D?)

Auditing

  • Official auditing of the course (i.e. taking the course for an Audit grade) is not permitted this semester.

  • Please email Joshmin if you need further clarification.

Students with Course Conflicts

Students with timing conflicts (i.e., who have another class offered at the same time) will be permitted to take this course.

However, there may be occasional days when we need you to arrive in person during class time (e.g. for student presentations). We will let you know of any dates we require you to be available, as soon as we know them.

Accommodations for Students with Disabilities

If you have a disability and have an accommodations letter from the Disability Resources office, please discuss your accommodation needs with Joshmin or one of the instructors as early in the semester as possible. We will work with you to ensure that accommodations are provided as appropriate.

If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, we encourage you to contact them at access@andrew.cmu.edu.

Take care of yourself (Read this!)

Do your best to maintain a healthy lifestyle this semester by eating well, exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. This will help you achieve your goals and cope with stress. All of us benefit from support during times of struggle. You are not alone. There are many helpful resources available on campus and an important part of the college experience is learning how to ask for help. Asking for support sooner rather than later is often helpful.

If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website at www.cmu.edu/counseling. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help.

Listeners outside CMU

Please feel free to reuse any of these course materials that you find of use in your own courses.

We ask that you retain any copyright notices, and include a written notice indicating the source of any materials you use.