CS762 Advanced Deep Learning

Course description: The course advances students’ knowledge in deep learning and enables exploring methods and applications of deep learning. The course covers cutting-edge topics, including neural architecture design, robustness and reliability of deep learning, learning with less supervision, lifelong machine learning, deep generative modeling, theoretical understanding of deep learning, and interpretable deep learning. It assumes that students already have a basic understanding of deep learning, familiarity with linear algebra, probability, statistics, and optimization, and proficiency at programming in Python.

Number of credits associated with the course: 3

How credit hours are met by the course: This class meetings for two, 75-minute class periods each week over the semester and carries the expectation that students will work on course learning activities (reading, writing, projects, etc.) for about 3 hours out of the classroom for every class period.

Prerequisite: COMP SCI 760 Machine Learning, Graduate/Professional Standing.

Time: TR 4:00PM - 5:15PM

Instruction Mode: Face to face

• Understand and be able to apply key elements and methods in deep learning (network architecture, training, backpropagation, stochastic gradient descent);
• Discuss and explain cutting-edge topics in AI and deep learning, including Transformers and LLMs, AI safety, model interpretability, foundation models, lifelong machine learning, and generative models.
• Identify, devise and apply deep learning approaches to a real-world problem;
• Identify and participate in original research-oriented project in a collaborative team;
• Search for sources of information and evaluation methods and datasets relevant to the project;
• Implement algorithms and write programs for training neural networks using deep learning libraries and frameworks;
• Develop analytic and problem-solving skills using computational approaches.
• Compose a final report that outlines the problem and your proposed method, evaluations, and/or theory.

In the regular lecture time (Tuesday and Thursday 4:00-5:15pm CT), we will have synchronous classes in person, during which the instructors will lecture or students will present, the class will enagage in Q&A, quizzes, and discussions.

We will use Piazza for Q&A. Please follow these rules:

• Please check if someone has posted the same / similar question before you; it’s much easier if we build on the thread.
• Use an informative Summary line to help others.

• Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville.
• Dive into Deep Learning, Aston Zhang and Zachary C. Lipton and Mu Li and Alexander J. Smola.

The grading for the course will be be based on the following. There will be no midterm or final exams.

• In-class quizzes: 20% (you can skip up to 2 of them)
• Paper presentation: 20%
• Project proposal: 10%
• Final project presentation: 10%
• Final project report (written): 40%

In-class quizzes

This will test students’ knowledge and understanding of the pre-class assignments (readings). Students are allowed one attempt at each quiz. The deadline to complete the in-class quiz is 4:15 PM the day of class. Each student is allowed to skip 2 in-class quizzes throughout the semester. The final grade for the quiz will be the average among all quizzes (deducted by the lowest 2 attempts).

Paper presentation

This will beevaluated on 4 technical aspects: (1) depth of content, (2) accuracy of content, (3) paper criticism, (4) discussion faciliation; as well as soft presentation skills including time management, responsiveness to audience, organization and presentation aids. Here is a great guide by Kayvon Fatahalian on how to give an effective academic talk. Please follow the tips when preparing for the paper presentation and final project presentation.

Project proposal

Groups will form early in the semester to begin the important process of team-building and topic identification. Projects should be done in groups of 3-4 people. We encourage students to find projects that relate to their ongoing research. A template is provided for proposal writing. Please email the project proposal to the TA by the proposal deadline. Each group submits only one proposal. The project proposal should include names of the students in the group, the research topic and problem, a brief description of tentative plan for the project.

Project policies

The following policies are adapted from Mark Craven's CS760 Fall 2016 Course and David Page's CS760 Spring 2018 Course. The project report will be due (pdf report and submission of any code written) to the TA by email, by the project deadline. Deadlines will be posted on the website (and please submit the proposal and the final report to the TA rather than on Canvas).

The basis for the project grade will be your written report and presentation. In particular, grading will NOT take into account the number of students in the group and depend only on the final outcome of the project. At the end of the semester, students will complete a grading rubric to rate the effort and involvement of their other team members in the final project. This will accordingly adjust the student's final grade for the sum of all the team project deliverables. The report should be in the style of a conference paper (e.g., using the style files of ICML'20), providing an introduction/motivation, discussion of related work, a description of your work that is detailed enough that the work could be replicated, and a conclusion. The format of the description of your work will depend on the nature of your project. If it is an algorithm-based project, then the description should make clear the algorithm(s) implemented and provide experimental results. If it is an application project, the description should say which system was used, how the data (or any other materials used) were collected, what experimental methodology was employed, and some estimate of the quality of the experimental results (e.g. a 10-fold cross-validation accuracy estimate). If it is a theoretical project, then the project description should consist of detailed definitions, theorems, and proofs. Evaluation of the project report will be similar to reviewing a conference/journal paper. See here for reviewing guidelines of a machine learning conference that describe well what we regard as great research outcomes. Details rubrics will be posted on Piazza.

There will be no midterm or final exam.