Jue Guo
← All courses

CSE 676 · Graduate

Deep Learning

Deep learning algorithms learn multi-level representations of data, with each level explaining the data at a higher level of abstraction. This course builds them from the ground up: the probability and linear models underneath, the networks and how to train them, the architectures that made the field, and the generative models at its frontier.

The course follows Bishop & Bishop's Deep Learning: Foundations and Concepts, one chapter per module. We start from probability, single-layer networks, and the mechanics of training — gradient descent, backpropagation, and regularization — then study convolutional networks, transformers and language models, and graph neural networks, and finish with sampling, latent-variable models, and the four families of deep generative models: GANs, normalizing flows, variational autoencoders, and diffusion models.

Every module has a full set of notes, linked from the outline below: the ideas and derivations in plain language, code with its real output, figures, and exercises. The early modules build layers, optimizers, and backpropagation from scratch in NumPy and check them against PyTorch. The later network modules (09–10, 12–13, 17–20) work in PyTorch, writing the key pieces ourselves; the probabilistic modules (11, 14–16) stay in NumPy, apart from the energy-based model of module 14, which is trained in PyTorch.

What the course covers

Fifteen weeks, one or two chapters of Bishop & Bishop a week, with the midterm in week 7 and the final in week 15. The weights below are those of Fall 2024, which was taught from Dive into Deep Learning with research-paper readings.

  1. Foundations
  2. Week 1 The deep learning revolution; probabilitiesA first model fitted end to end, overfitting and model selection; Bayes' theorem, the Gaussian, change of variables, and information theory.
  3. Week 2 Standard distributions; single-layer networks: regressionThe distributions networks output and the multivariate Gaussian, with histograms, kernels, and nearest neighbors; linear regression as a one-layer network, its geometry, and the bias–variance trade-off.
  4. Week 3 Single-layer networks: classification; deep neural networksLogistic and softmax regression, ROC curves; multilayer networks, activation functions, error functions, and mixture density networks.
  5. Training deep networks
  6. Week 4 Gradient descent; backpropagationSGD, momentum, Adam, initialization, and normalization, all built from scratch; backpropagation and automatic differentiation.
  7. Week 5 RegularizationInductive bias and equivariance, weight decay, early stopping and double descent, residual connections, ensembles, and dropout.
  8. Architectures
  9. Week 6 Convolutional networksConvolution, pooling, and architectures; saliency and adversarial examples; detection, U-net segmentation, and style transfer.
  10. Week 7 Midterm examCovers modules 01–10.
  11. Week 8 Structured distributionsDirected graphical models, conditional independence and d-separation, and sequence models.
  12. Week 9 TransformersAttention, transformer layers, tokenization, RNNs, a small language model trained from scratch, and vision transformers.
  13. Week 10 Graph neural networksPermutation equivariance, message passing and GCNs, graph attention, over-smoothing, and equivariant networks.
  14. Sampling, latent variables, and generative models
  15. Week 11 Sampling; discrete latent variablesRejection and importance sampling, MCMC, and Langevin dynamics for energy-based models; K-means, Gaussian mixtures, EM, and the evidence lower bound.
  16. Week 12 Continuous latent variables; generative adversarial networksPCA, probabilistic PCA, factor analysis, ICA, and the Kalman filter, and why nonlinear latent-variable models need new methods; the adversarial game, its training problems, and CycleGAN.
  17. Week 13 Normalizing flows; autoencodersCoupling, autoregressive, and continuous flows; denoising and masked autoencoders, and variational autoencoders.
  18. Week 14 Diffusion modelsThe noising process, learning to denoise, score matching and SDEs, and classifier-free guidance.
  19. Week 15 Review and final examThe final exam is cumulative, with emphasis on modules 11–20.

How the course runs

Two lectures a week; office hours are announced at the start of the term, and the schedule may shift with the pace of the class. Each week works through one or two chapters. Read the module's notes, and ideally the matching chapter, before class. The exercises at the end of each module are the best preparation for the assignments and the exams: try each derivation yourself before checking it against the notes, and change the code to see what breaks.

  • Attendance (10%) — random attendance checks.
  • Programming assignments (30%) — two assignments in PyTorch, each with a few weeks to complete; release and due dates are announced in class.
  • Midterm exam (30%) — in week 7, covering modules 01–10, with a week's notice of the exact date.
  • Final exam (30%) — in week 15, cumulative, with emphasis on modules 11–20.

Grades use an absolute scale with no curve: 95–100 A, 90–94 A−, 85–89 B+, 80–84 B, 75–79 B−, 70–74 C+, 65–69 C, 60–64 C−, 55–59 D, below 55 F.

Getting set up

The code in the notes needs Python 3.10 or later with NumPy 2, SciPy, PyTorch 2 with torchvision, and networkx (module 13 only). Two data sets are downloaded the first time they are used, into a data/ folder next to your notebook: MNIST through torchvision (about 12 MB; modules 06–10, 12, 15–17, and 19) and the Tiny Shakespeare text (about 1 MB; modules 11–12). A laptop CPU runs every example: the models are deliberately small and trained on subsets of MNIST or on synthetic data, and each module runs in a few seconds to a minute or two. A free Google Colab notebook works too. Paste the cells of a module into a notebook in order and run them. Every random number comes from a seeded generator, and the notes were run on one CPU thread (torch.set_num_threads(1)), so your output should match closely; with several threads or on a GPU, the last digits of training numbers can differ, and timings always will.

Course policies

Collaboration

Discussing the modules and the assignments with classmates at the level of ideas is encouraged, but your code and write-ups must be your own. Helping someone debug an error message is fine; sharing code or results is not.

AI and LLM use

  • Allowed without disclosure: autocomplete-style help (syntax lookups, translating code between libraries), debugging individual error messages, and reading recommendations.
  • Allowed with disclosure: explanations of mathematical concepts you are stuck on, and suggested approaches after you have thought the problem through. A one-line note in your submission is enough.
  • Not allowed: generating programming-assignment solutions or large blocks of assignment code or text with an LLM. Undisclosed LLM-generated work is an academic-integrity violation.

When in doubt, disclose. Disclosure costs you nothing; non-disclosure can cost you the course.