Schedule for CMU 10-405/10-605/10-805

Date	Class Type	Topic	Resources	Announcements
Mon Aug 24, 2026	Lecture	Overview / Slides (pdf) / Slides (pptx)	Resources Matt Gormley's discussion of Slot A/Slot B process William W. Cohen (1993). Efficient pruning methods for separate-and-conquer rule learning systems Banko, Michele, and Eric Brill (2001). Scaling to very very large corpora for natural language disambiguation Google NGrams Viewer Norvig, Pereira, Halevy (2009). The Unreasonable Effectiveness of Data Yoshua Bengio (2009). Learning Deep Architectures for AI code for doing ngram queries Hoffman et al (2022). Training Compute-Optimal LLMs	HW 1 out - Scalable Entity Resolution and Naive Bayes
Wed Aug 26, 2026	Lecture	Map-Reduce and Spark / Slides (pdf) / Slides (pptx)	Resources Visualizing the cost of operations Historical cost of storage code for Hazsoup - a more readable minimal implementation of Map-Reduce Unix for Poets, Ken Church, 1994 MapReduce, Dean and Shemawat, 2004 Visualize merge sorts Visualize shuffle sort/partition Polars Getting Started Tutorial
Fri Aug 28, 2026	Recitation	Recitation 1: Python Polars	Resources Polars Essentials (part 2) Colab Notebook Colab Notebook Solutions Slides
Mon Aug 31, 2026	Lecture	Workflows for Map-Reduce Systems / Slides (pdf) / Slides (pptx)	Resources A Language Model Approach to Keyphrase Extraction, Tomokiyo and Hurst, 2003 Phrase extraction polars code k-means polars code matrix multiplication polars code
Wed Sep 2, 2026	Lecture	Learning as Optimization 1 / Slides (pdf) / Slides (pptx)	Resources Previous class lecture on Sept 11 2024 pagerank polars code
Fri Sep 4, 2026	Recitation	Recitation 2: Linear Algebra Review	Resources Slides Notebook
Mon Sep 7, 2026	No Class	Labor Day
Wed Sep 9, 2026	Lecture	Learning as Optimization 2 - Distributed learning, parameter mixing, kernels, all-reduce / Slides (pdf) / Slides (pptx)	Resources Previous class lecture on Sept 16 2024	HW 1 Slot A due HW 2 out - Parallel Linear Regression
Fri Sep 11, 2026	Recitation	Recitation 3: PyTorch HW1 Writing Session	Resources Notebook	HW 1 Slot A Grading deadline: Sat Sept 12
Mon Sep 14, 2026	Lecture	Learning as Optimization 3 - sparsity, hash kernels, matrix factorization / Slides (pdf) / Slides (pptx)	Resources Agarwal et al, A Reliable Effective Terascale Linear Learning System William's notes on SGD for Logistic Regression and Sparsity Hash Kernels, PMLR 2009 Feature hashing for large scale multitask learning, ICML 2009 Hogwild, A Lock-Free Approach to Parallelizing Stochastic Gradient Descent, Rech et al, 2011 FastText classification library Bag of Tricks for Efficient Text Classification, Joulin et al 2016 Communication-Efficient Distributed Deep Learning, Tang et al 2023 Distributed Training Strategies for the Structured Perceptron, McDonald et al, 2010 Large-scale matrix factorization with distributed stochastic gradient descent - Gemulla et al 2011
Wed Sep 16, 2026	Lecture	Randomized Algorithms 1 - Bloom filters and Count-Min sketches / Slides (pdf) / Slides (pptx)	Resources Agarwal et al, A Reliable Effective Terascale Linear Learning System William's Notes on Randomized Algorithms code for Bloom Filter Short and Deep: Sketching and Neural Networks, Daniely et al Sketch Algorithms for Estimating Point Queries in NLP, 2012 Previous course lecture, 9/25/2024	HW1 Slot B due
Fri Sep 18, 2026	Recitation	Recitation 4: Probability, Evaluation Metrics and Minhashing
Mon Sep 21, 2026	No Class	Yom Kippur
Wed Sep 23, 2026	Lecture	Randomized Algorithms 2 - Locality Sensitive Hashing, Minhash and Simhash / Slides (pdf) / Slides (pptx)	Resources Fast Exact Search in Hamming Space with Multi-Index Hashing, 2014 Accelerating Large-Scale Inference with Anisotropic Vector Quantization, 2020 The FAISS library, 2024	HW 2 slot A due HW 3 out - Locality-Sensitive Hashing
Fri Sep 25, 2026	Recitation	HW2 Writing Session		HW 2 slot A grading deadline: Sept 26
Mon Sep 28, 2026	Lecture	AutoDiff / Slides (pdf) / Slides (pptx)	Resources A simple explanation of reverse-mode automatic differentiation, Justin Domke code for autodiff with Wengert lists
Wed Sep 30, 2026	Lecture	Parallel Optimization with GPUs / Slides (pdf) / Slides (pptx)	Resources Previous class lecture on Nov 4 2024 Previous class lecture on Oct 21 2024	HW 3 Slot A due
Fri Oct 2, 2026	Recitation	HW3 Writing Session		HW 3 Slot A grading deadline: Oct 4
Mon Oct 5, 2026	Lecture	Exam Review and QA / Slides (pdf) / Slides (pptx)		HW 3 Slot B due
Wed Oct 7, 2026	Lecture	Exam 1
Fri Oct 9, 2026	Recitation	No Recitation		Miniproject: 10-605 students apply to lead a miniproject
Mon Oct 12, 2026	No Class	Fall Break
Wed Oct 14, 2026	No Class	Fall Break
Fri Oct 16, 2026	No Class	Fall Break
Mon Oct 19, 2026	Lecture	Deep Learning: Background and Architectures / Slides (pdf) / Slides (pptx)	Resources online book on deep networks Understanding the difficulty of training deep feedforward neural networks, Glorot and Bengio, AIStats 2010 Convolution demo Deep Residual Learning for Image Recognition, He et al, CVPR 2016 Efficient Estimation of Word Representations in Vector Space, Mikolov et al, 2013 Visualize Word2Vec Karpathy blog post on LSTMs NMT by jointly learning to align and translate, 2016	HW 4 out - LoRA
Wed Oct 21, 2026	Lecture	Deep Learning: Transformers / Slides (pdf) / Slides (pptx)		Miniproject: open-ended miniproject proposals due
Fri Oct 23, 2026	Recitation	Recitation: PyTorch / HW4 / Notebook
Mon Oct 26, 2026	Lecture	Hyperparameter Search / Slides (pdf) / Slides (pptx)	Resources PDF of Greg's slide deck Hoffman et al (2022). Training Compute-Optimal LLMs (Mis)fitting - a study of scaling laws, ICLR 2025 Hyperband - A Novel Bandit Based Approach to Hyperparameter Optimization, JMLR 2018	Miniproject: feedback on open-ended miniproject proposals given; proposals are posted for class
Wed Oct 28, 2026	Lecture	Model compression / Slides (pdf) / Slides (pptx)	Resources Distilling the Knowledge in a Neural Network, Hinton, Vinyals, Dean, 2015 Model Compression, Bucila et al, 2006 Raschka blog post, 2023 Lei Mao blog post, 2023 LLM.int8(), Dettmers et al 2022 Dettmers and Zettlemoyer 2023 PV-Tuning, Malinovskii et al, 2024 The cat neuron paper Learning to Generate Reviews and Discovering Sentiment, Radford et al 2017 To prune or not to prune ..., Zhu and Gupta 2017 A simple and effective approach ... (the Wanda paper), Sun et al Accelerating Sparse Deep Neural Networks, Mishra et al 2021 LLM Pruner, 2023 Shortened Llama, ... Kim et al 2024 Sheared LLama ... , Xia et al, 2024 Han et al 2016
Fri Oct 30, 2026	Recitation	Recitation - HW5
Mon Nov 2, 2026	Lecture	Embarrassingly Parallel Training / Slides (pdf) / Slides (pptx)	Resources Branch-train-merge paper Branch-train-mix paper Levy and Goldberg demystification of word2vec (1/2) 2014 Levy and Goldberg demystification of word2vec (2/2) 2014 Task arithmetic paper, 2023 Weight disentanglement and neural tangent kernel paper, 2023 TIES-merging paper, 2023 LoRA hub paper, 2024 PHATGOOSE paper, 2024	HW 4 Slot A due HW 5 out - GPT-2 Training Miniproject: Structured miniproject details released, project teams formed, and declaration form due
Wed Nov 4, 2026	Lecture	KV-Caching for LLMs / Slides (pdf) / Slides (pptx)	Resources ETC: Encoding Long and Structured Inputs in Transformers, Ainsle et al 2020 Big Bird: Transformers for Longer Sequences, Zaheer et al, 2020 Reformer: the Efficient Transformer, Kitaev et al, 2020 H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, Zhang et al 2023 Efficient streaming LMs with attention sinks, Xiao et al 2024 SnapKV: LLM Knows What You are Looking for Before Generation, Li et al 2024	Miniproject: structured miniproject survey due
Fri Nov 6, 2026	Recitation	Recitation - Structured miniproject / Slides (pdf)		HW 4 Grading deadline: Nov 5
Mon Nov 9, 2026	Lecture	KV Caching and Model Compression Recap / Slides (pdf) / Slides (pptx)	Resources SnapKV: LLM Knows What You are Looking for Before Generation, Li et al 2024 Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, Ge et al 2023 Cross-Layer Attention paper, NeurIPS 2024 Mooncake paper, usenix 2025	HW4 Slot B Deadline
Wed Nov 11, 2026	Lecture	Contrastive learning and retrieval	Resources DeepHash paper, 2017 DPR paper, 2020 Condenser paper EntityQuestions paper MoCo paper
Fri Nov 13, 2026	Recitation	HW4 Writing Session
Mon Nov 16, 2026	Lecture	RAG / Optimizing Transformer Architectures	Resources RAG for knowledge-intensive NLP tasks, 2020 REALM paper, 2020 Fusion in decoder paper, 2021 FiDO paper Shazeer paper on optimizing decoder inference with multihead queries Parallel context windows, 2023 Block-attention for RAG, 2024 Dynamic block-sparse attention for ICL, 2025 TurboRAG paper HyDE paper LameR paperx Echo embedding paper PromptEOL paper ExpandR paper DPO paper Michiel's talk (Google slides) Michiel's talk (PDF) Shazeer paper analyzing operational intensity LUMEN paper GLIMMER paper	HW5 Slot A Due
Wed Nov 18, 2026	Lecture	Guest Lecture - Leo Boytsov - Robustness, Generalization, and Practical Limits of Deep Learning, A Practitioner’s Perspective		Miniproject: project check in due (written report OR oral check-in - see miniproject handouts).
Fri Nov 20, 2026	Recitation	HW5 Writing Session		HW5 Slot A grading deadline
Mon Nov 23, 2026	Lecture	Network Effects and Scalable Evaluation / Slides (pdf) / Slides (pptx)	Resources Information cascades, Ch 16, Easly and Kleinberg, 2010 Salganic, Dodds, Watts, Science 2006 Schelling Spatial Segregation model Prediction Powered Inference, Angelopolous 2023, Science Bayesian PPI, Hofer et al 2024 Kamaloo et al, 2023 Fisch et al, 2024	HW5 slot B deadline
Wed Nov 25, 2026	No Class	Thanksgiving
Fri Nov 27, 2026	No Class	Thanksgiving
Mon Nov 30, 2026	Lecture	Exam Review and QA / Slides (pdf) / Slides (pptx)		Miniprojects due
Wed Dec 2, 2026	Lecture	Exam 2

10-405/10-605/10-805: ML with Large Datasets, Fall 2026

Schedule