CSE 240D

This is the anticipated schedule for the course. The schedule is subject to change. Each lecture lists its assigned readings and, where applicable, optional readings.

Each student will present one paper of their choosing. Students who sign up for 04/06 will receive extra credit. More papers will be announced if needed.

#	Lecture	Date	Slides	Presenters
1	Introduction and Dark Silicon Assigned readings: Esmaeilzadeh et al., "Dark Silicon and the End of Multicore Scaling", ISCA 2011	03/30/2026	Intro and Dark Silicon	Hadi
2	Scaling Laws Assigned readings: Kaplan et al., "Scaling Laws for Neural Language Models", 2020 Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla), NeurIPS 2022 Goodfellow, Bengio & Courville, "Deep Learning" Chapter 8: Optimization for Training Deep Models, 2016 He et al., "Deep Residual Learning for Image Recognition" (ResNet), CVPR 2016 Kingma & Ba, "Adam: A Method for Stochastic Optimization", ICLR 2015	04/01/2026	Scaling Laws	Hanyang
3	Learning Paradigms + Alignment Assigned readings: Esmaeilzadeh et al., "Dark Silicon and the End of Multicore Scaling", ISCA 2011 (25-Year Retrospective) (no critique required) Goodfellow, Bengio & Courville, "Deep Learning" Chapters 5 + 15: Supervised/Unsupervised Learning, 2016 Radford et al., "Improving Language Understanding by Generative Pre-Training" (GPT-1), 2018 Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT/RLHF), NeurIPS 2022 DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", 2025 Optional readings: Alammar, "The Illustrated GPT-2 (Visualizing Transformer Language Models)", 2019	04/06/2026	GPT-1, InstructGPT/RLHF, DeepSeek-R1	Ethan Jenkins, Yuvanand Saravanan, Hantian Lin
—	Learning Paradigms + Alignment (continued)	04/08/2026
—	Gradient Descent / Backpropagation Assigned readings: Rumelhart, Hinton & Williams, "Learning Representations by Back-Propagating Errors", Nature 1986	04/13/2026		Hadi
4	Attention, Transformers + Diffusion Assigned readings: Vaswani et al., "Attention Is All You Need", NeurIPS 2017 Touvron et al., "LLaMA: Open and Efficient Foundation Language Models", 2023 Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models" (Stable Diffusion), CVPR 2022 Peebles & Xie, "Scalable Diffusion Models with Transformers" (DiT), ICCV 2023 Optional readings: Alammar, "The Illustrated Transformer", 2018 Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units" (BPE), ACL 2016 Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (RoPE), 2021	04/15/2026	Attention, DiT	Hadi, Hanyang
—	Attention, Transformers + Diffusion (continued)	04/20/2026	Attention	Hadi, Hanyang
5	Model Capabilities: MoE, Multimodal, Reasoning Assigned readings: Liu et al., "Visual Instruction Tuning" (LLaVA), NeurIPS 2023 Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022	04/22/2026	LLaVA, Chain-of-Thought	Tejus Singh, Sankalpa Hota
6	Model Capabilities (continued) + Agentic Systems Assigned readings: Hao et al., "Training Large Language Models to Reason in a Continuous Latent Space" (Coconut), ICLR 2025 Packer et al., "MemGPT: Towards LLMs as Operating Systems", 2023	04/27/2026	Agents, Coconut, MemGPT	Hanyang, Wentao Chen, Chi-Han Chiu
7	Distributed Training Assigned readings: Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", 2019 + Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM", SC 2021 (one presentation) Meta, "Scaling Llama 3 Training with Efficient Parallelism Strategies", ISCA 2025	04/29/2026	Megatron-LM, Llama 3 Training	Aditya Kamat, Nikhita Neelakanta
8	Efficient Inference Architectures Assigned readings: Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", EMNLP 2023 Optional readings: Leviathan et al., "Fast Inference from Transformers via Speculative Decoding", ICML 2023 Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (MQA), 2019 Lieber et al., "Jamba: A Hybrid Transformer-Mamba Language Model", 2024	05/04/2026	GQA	Sanskriti Goyal
9	Model Compression: Quantization, LoRA, Sparsity Assigned readings: Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023 Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", ICLR 2022 Mishra et al., "Accelerating Sparse Deep Neural Networks", 2021	05/06/2026	GPTQ, LoRA, Sparse DNNs	Shivam Yadav, Hanyang, Mohamed Ibrahim
—	Model Compression: Quantization, LoRA, Sparsity (continued) Optional readings: DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" (MLA), 2024 Chang et al., "Palu: Compressing KV-Cache with Low-Rank Projection", ICLR 2025 Yuan et al., "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention" (NSA), 2025	05/11/2026
10	GPU Architecture + FlashAttention / Kernel Optimizations Assigned readings: Luo et al., "Benchmarking and Dissecting the Nvidia Hopper GPU Architecture", IPDPS 2024 Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", NeurIPS 2022 + Dao, "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning", ICLR 2024 (one presentation) Optional readings: Shah et al., "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision", NeurIPS 2024 Zadouri et al., "FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling", 2026 Huerta et al., "Analyzing Modern NVIDIA GPU cores", 2025	05/13/2026	Hopper GPU, FlashAttention	Amaan M., Nisha
11	Domain-Specific Accelerators: TPUs, Groq, PIM Assigned readings: Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit", ISCA 2017 + Norrie et al., "The Design Process for Google's Training Chips: TPUv2 and TPUv3", IEEE Micro 2021 (one presentation) Abts et al., "Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads" (Groq), ISCA 2020 Liang et al., "PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference", ASPLOS 2025	05/18/2026	TPU	John
—	Domain-Specific Accelerators: TPUs, Groq, PIM (continued)	05/20/2026	Groq	Shoumik, TBD
—	NO CLASS — Memorial Day	05/25/2026
12	Compiler Optimizations Assigned readings: Chen et al., "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning", OSDI 2018 Lattner et al., "MLIR: A Compiler Infrastructure for the End of Moore's Law", 2020 Optional readings: Ouyang et al., "KernelBench: Can LLMs Write Efficient GPU Kernels?", 2025	05/27/2026	TVM, MLIR	Jiahui Cheng, Abhinandan Sharma
13	KV Cache Management + Prefill-Decode Scheduling Assigned readings: Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023 Qin et al., "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving", FAST 2025 Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", OSDI 2024	06/01/2026	Mooncake	Shan Ananth
14	Network Infrastructure + DPUs Assigned readings: Xiang et al., "ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching", 2025 Optional readings: Gangidi et al., "RDMA over Ethernet for Distributed Training at Meta Scale", ACM SIGCOMM 2024	06/03/2026	ShadowServe	TBD

CSE 240D: System Design for Machine Intelligence (Spring 2026)

Instructor: Hadi Esmaeilzadeh

Email: hadi [AT] ucsd [DOT] edu

Office: CSE 3228

Office hours: by appointment

Class time: MW 6:30–7:50 PM, EBU3B 4140

TA: Hanyang Xu

Email: hax032 [AT] ucsd [DOT] edu

Office: CSE 3254

Office hours: by appointment