Schedule for CMU 10-405/10-605

Date	Class Type	Topic	Resources	Announcements
Mon Jan 13, 2025	Lecture	Overview / Slides (pdf) / Slides (pptx) / Recording	William W. Cohen (1993). Efficient pruning methods for separate-and-conquer rule learning systems Banko, Michele, and Eric Brill (2001). Scaling to very very large corpora for natural language disambiguation Google NGrams Viewer Norvig, Pereira, Halevy (2009). The Unreasonable Effectiveness of Data Yoshua Bengio (2009). Learning Deep Architectures for AI Sample code from lecture for doing ngram queries Hoffman et al (2022). Training Compute-Optimal LLMs
Wed Jan 15, 2025	Lecture	Map-Reduce and Spark / Slides (pdf) / Slides (pptx) / Recording	Visualizing the cost of operations Historical cost of storage Bash Reduce - minimal implementation of Map-Reduce	HW 1 out - Entity Resolution and Naive Bayes in Spark
Fri Jan 17, 2025	Recitation	Recitation: PySpark / Slides (pdf) / Notebook / Recording
Wed Jan 22, 2025	Lecture	Workflows for Map-Reduce Systems / Slides (pdf) / Slides (pptx) / Recording	Ken Church (1994). Unix for Poets Demo code from lecture (all in one file) Code from lecture for doing wordcounts in Spark Code for phrase-finding using Spark A Language Model Approach to Keyphrase Extraction Code for slow implementation of PageRank Faster implementation of PageRank Fastest implementation of PageRank (v2)
Fri Jan 24, 2025	Recitation	Recitation: Linear Algebra Review / Slides (pdf) / Notebook / Recording
Mon Jan 27, 2025	Lecture	Learning as Optimization 1 / Slides (pdf) / Slides (pptx) / Recording	Previous class lecture on Sept 9 2024 Previous class lecture on Sept 11 2024
Wed Jan 29, 2025	Lecture	Learning as Optimization 2 / Slides (pdf) / Slides (pptx) / Recording	Previous class lecture on Sept 16 2024 William's notes on SGD for Logistic Regression and Sparsity Hash Kernels, PMLR 2009 Feature hashing for large scale multitask learning, ICML 2009	HW 2 out - Parallel Linear Regression in Spark HW 1 due
Fri Jan 31, 2025	Recitation	No Recitation
Mon Feb 3, 2025	Lecture	Learning as Optimization 3 / Slides (pdf) / Slides (pptx) / Recording	Hogwild, A Lock-Free Approach to Parallelizing Stochastic Gradient Descent, Rech et al, 2011 FastText classification library Bag of Tricks for Efficient Text Classification, Joulin et al 2016 Communication-Efficient Distributed Deep Learning, Tang et al 2023 Distributed Training Strategies for the Structured Perceptron, McDonald et al, 2010 Large-scale matrix factorization with distributed stochastic gradient descent - Gemulla et al 2011
Wed Feb 5, 2025	Lecture	Randomized Algorithms 1 - Bloom Filters and Count-Min Sketches / Slides (pdf) / Slides (pptx) / Recording	William's Notes on Randomized Algorithms Demo Bloom filter implementation in Python {'Short and Deep': 'Sketching and Neural Networks, Daniely et al'} Code from lecture for bloom filter implementation Sketch Algorithms for Estimating Point Queries in NLP - Goyal et al 2012 Previous course lecture, 9/25/2024 Sketch Algorithms for Estimating Point Queries in NLP, 2012
Fri Feb 7, 2025	Recitation	Recitation: Probability for LSH / Slides (pptx) / Recording
Mon Feb 10, 2025	Lecture	Randomized Algorithms 2 - Locality Sensitive Hashing / Slides (pdf) / Slides (pptx) / Recording	Fast Exact Search in Hamming Space with Multi-Index Hashing, 2014 Accelerating Large-Scale Inference with Anisotropic Vector Quantization, 2020 The FAISS library, 2024	HW 3 out - Locality-Sensitive Hashing
Wed Feb 12, 2025	Lecture	Countmin Application / AutoDiff / Slides (pdf) / Slides (pptx) / Recording	Faithful KB Embeddings - Sun et al, 2020 A simple explanation of reverse-mode automatic differentiation, Justin Domke Sample code for autodiff with Wengert lists Sample code for checking autodiff implementation with PyTorch
Fri Feb 14, 2025	Recitation	Recitation: AWS / Slides (pptx) / Recording
Mon Feb 17, 2025	Lecture	Parallel Optimization with GPUs 1 / Slides (pdf) / Slides (pptx) / Recording	Mythbusters explain GPUs GFlops / dollar up to 2022 Benchmarking GPUs in PyTorch on a Mac Fall 2024 10-605 lecture on ML hardware Code for comparing GPU vs CPU times in PyTorch
Wed Feb 19, 2025	Lecture	Parallel Optimization with GPUs 2 / Slides (pdf) / Slides (pptx) / Recording	Previous class lecture on Nov 4 2024 Previous class lecture on Oct 21 2024
Fri Feb 21, 2025	Recitation	Recitation: Practice Exam		HW 3 due
Mon Feb 24, 2025	Lecture	Exam Review and QA
Wed Feb 26, 2025	Lecture	Exam 1
Mon Mar 10, 2025	Lecture	Deep Learning: Background and Architectures		HW: LoRA
Wed Mar 12, 2025	Lecture	Deep Learning: Architectures and Param Efficient Tuning
Fri Mar 14, 2025	Recitation	Recitation: PyTorch / HW4
Mon Mar 17, 2025	Lecture	Guest lecture - Greg Kochanski, Google Deep Mind
Wed Mar 19, 2025	Lecture	Deep Learning: Scaling Transformers
Fri Mar 21, 2025	Recitation	Recitation - HW5
Mon Mar 24, 2025	Lecture	Guest lecture - Haitian Sun, Apple		HW: GPT-2 Training and Miniproject released
Wed Mar 26, 2025	Lecture	Scaling Transformers / Hyperparameter Search and Scaling Laws
Fri Mar 28, 2025	Recitation	Recitation - Miniproject
Mon Mar 31, 2025	Lecture	Contrastive learning
Wed Apr 2, 2025	Lecture	Retrieval Augmented Generation 1
Mon Apr 7, 2025	Lecture	Retrieval Augmented Generation 2		Previous HW due
Wed Apr 9, 2025	Lecture	Parameter Servers
Fri Apr 11, 2025	Recitation	No Recitation
Mon Apr 14, 2025	Lecture	Scalable Graph Processing 1
Wed Apr 16, 2025	Lecture	Scalable Graph Processing 2
Fri Apr 18, 2025	Recitation	Recitation - Practice Exam
Mon Apr 21, 2025	Lecture	Exam Review and QA		Miniproject due
Wed Apr 23, 2025	Lecture	Exam 2

10-405/10-605: ML with Large Datasets, Spring 2025

Schedule (subject to change)