Materials and Resources
Textbooks, one paper per week, derivations, toolchain
Core Textbooks
| Code | Source | Used for |
|---|---|---|
| JM3 | Jurafsky & Martin, Speech and Language Processing, 3rd ed. online draft, 2026-08-19 release (https://web.stanford.edu/~jurafsky/slp3/) | The teaching baseline for the whole course. This release is reorganized into three volumes and 26 chapters, with Volume I devoted to LLMs |
| RAS | Raschka, Build a Large Language Model (From Scratch), Manning, 2024 (a Traditional Chinese edition exists) | The primary text for building components by hand. The labs in W3, W4 and W8 take their skeletons from it |
| RAS-R | Raschka, Build a Reasoning Model (From Scratch), Manning | Hand-built material for W9 and W10 — the DPO loss, verifiers, budget forcing |
| XZ | Xiao & Zhu, Foundations of Large Language Models, arXiv:2501.09223 (2025-01, last revised 2025-06) | An arXiv monograph, not a published book. Treat it as free lecture notes rather than as a textbook |
| LLMBook | 趙鑫、李軍毅、周昆、唐天一、文繼榮, 大語言模型, 高等教育出版社, 2024 (13 chapters) | The Chinese companion. Its site carries lecture PDFs, code and the original slides on request |
| BB | Bishop & Bishop, Deep Learning: Foundations and Concepts, Springer, 2024 | Prerequisite mathematics and deep-learning background; free online edition |
| UDL | Prince, Understanding Deep Learning, MIT Press; GitHub v5.0.3, 2026-02-09, ch 1–21, CC BY-NC-ND | The best catch-up text for a class with no prerequisites: free PDF, notebooks, slides and an answer booklet, and still maintained |
| HOLLM | Alammar & Grootendorst, Hands-On Large Language Models, O’Reilly, 2024 (12 chapters) | A hands-on companion. The visualizations in ch 2 (tokens and embeddings) and ch 8 (RAG) are unusually good. Its notebooks are written for Colab, so the backend has to be swapped on the demo machine |
| ZZM | Zong, Zhao & Ma, Natural Language Processing and Large Language Models: Theory, Hands-on Codes, and Case Studies, Springer Nature Singapore / Tsinghua University Press, 2026 (ISBN 978-981-92-0681-0; open access, 400 pages) | The English companion for W1–W3. Ch 5 (n-grams, smoothing, perplexity, FNN- and LSTM-based LMs), ch 3 (word2vec, ELMo) and ch 4 (seq2seq through Transformer) line up almost section by section with W1 and W3; ch 2 is the shortest catch-up chapter for anyone arriving without the prerequisites; ch 7 is the only treatment of Chinese word segmentation on this list. Read it, do not run it — see the note below |
| LLMSurvey | Zhao et al., A Survey of Large Language Models, arXiv:2303.18223 (last revised 2026-03-18, 144 pages, 1081 references, marked ongoing) | A reference, not a textbook |
Three things to know about JM3
Checked directly against the Stanford page.
- It is still a draft with no print edition (the site’s own answer to “when will the book be finished?” is “don’t ask”). Download and keep the
2026-08-19release, because chapter numbers move between releases and this course cites that one. - The structure of the 2026-08-19 release — the largest improvement since the 2024 version, since pretraining and post-training finally have separate chapters:
- Vol I — Large Language Models: 1 Introduction (rewritten for this release) / 2 Words and Tokens / 3 N-gram LM / 4 Logistic Regression & Text Classification / 5 Embeddings / 6 Neural Networks / 7 Transformers and Pretraining / 8 Post-training
- Vol II — Advanced LLM Topics and Tools: 9 Masked LM / 10 Interpretability (incomplete) / 11 Information Retrieval and RAG / 12 Agents “[not written yet]” / 13 MT / 14 RNNs & LSTMs / 15–17 speech
- Vol III — Annotating Linguistic Structure: 18–26 (POS/NER, parsing, IE, SRL, coreference, discourse, conversation)
- W13 has no textbook — ch 12 Agents is unwritten, so that week runs entirely on papers and lecture notes. Ch 10 Interpretability is also incomplete; use it with care in W14.
Two limits on ZZM
- The code is on a different stack. Chapters 7–15 are written entirely in PaddlePaddle / PaddleNLP against Baidu’s ERNIE models and AI Studio: the book mentions
paddle474 times andpaddlenlp123 times, andtorchandpytorchexactly zero times. This course runs on Hugging Facetransformerswith Qwen3, so ZZM is reading, not lab material. - Coverage stops around 2023. Ch 6 compresses instruction tuning and RLHF into a section each, and the substance of W5–W14 — RoPE and long context, MoE and state-space models, scaling laws, DPO and GRPO, reasoning and test-time compute, inference efficiency, RAG, agents, evaluation and interpretability — is simply not in the book.
The authors are candid about the application chapters themselves: ch 1.2 says they demonstrate “the implementation process of the task through a specific example, rather than getting a practical system with the best performance.” Read the application chapters in that spirit.
Deliberately not assigned
Goldberg, Neural Network Methods for NLP (2017) and Paaß & Giesselbach, Foundation Models for NLP (2023) have both dated badly, and appear only as historical context in W1. Raaijmakers, Large Language Models (MIT Press, 2025) is conceptual and social rather than technical — not deep enough for a graduate course. Burkov, The Hundred-Page Language Models Book (2025) is too thin to carry the course, but works as pre-reading before week 1.
One Paper per Week
| Week | Topic | If you read only one |
|---|---|---|
| W1 | From n-grams to seq2seq | Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015 |
| W2 | Tokenization | Limisiewicz et al., Compute Optimal Tokenization, arXiv:2605.01188 |
| W3 | Attention | Vaswani et al., Attention Is All You Need, NeurIPS 2017 — with Jain & Wallace, Attention is not Explanation |
| W4 | The full block | Wortsman et al., Small-scale proxies for large-scale Transformer training instabilities, arXiv:2309.14322 |
| W5 | Position and long context | Liu et al., Lost in the Middle, TACL 2023 |
| W6 | Architecture routes | Du et al., Kimi Linear, arXiv:2510.26692 — with Dao & Gu, Mamba-2 |
| W7 | Pre-training and scaling | Hoffmann et al., Chinchilla, arXiv:2203.15556 — with Besiroglu et al.’s replication |
| W8 | SFT and PEFT | Shuttleworth et al., LoRA vs Full Fine-tuning: An Illusion of Equivalence, arXiv:2410.21228 |
| W9 | Alignment and RL | Rafailov et al., DPO, NeurIPS 2023 — with the GRPO section of Shao et al., DeepSeekMath |
| W10 | Reasoning and test-time compute | Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model?, arXiv:2504.13837 |
| W11 | Inference efficiency | Leviathan et al., Fast Inference from Transformers via Speculative Decoding, ICML 2023 |
| W12 | RAG | Li et al., RAG or Long-Context LLMs?, EMNLP 2024 industry track |
| W13 | Agentic systems | Cemri et al., Why Do Multi-Agent LLM Systems Fail?, arXiv:2503.13657 |
| W14 | Evaluation and interpretability | Korznikov et al., Sanity Checks for Sparse Autoencoders, arXiv:2602.14111 — with Miller, Adding Error Bars to Evals |
Why these. The table leans deliberately towards critical work and negative results rather than the paper that first proposed each method. The originals are largely in the textbook — JM3 Vol I covers the spine of W1–W9 — while “here is where this method breaks” has no textbook and is worth far more to a graduate student. If you read one paper for a week, read the critical one.
Derivations and Hand-Built Components
One board derivation and one component per week. Everything is demonstrated in class. The last column says what each component needs to run on, not how long it takes: the demo machine has no CUDA, and nothing here has been timed on it yet. Timings will be added once they are measured rather than estimated.
| Week | At the board | Built by hand | Runs on |
|---|---|---|---|
| W1 | Cross-entropy as compression; the power-law decay of unseen mass under Zipf–Heaps; the BPTT product and its singular-value bound | — (demos only) | CPU |
| W2 | Vocabulary size inside the scaling law | BPE training, encoding, decoding | CPU — tokenizers are hardware-independent |
| W3 | The variance of q^\top k → 1/\sqrt{d_k}; the KV-cache formula | Scaled dot-product attention + KV cache | CPU — the gradient check needs float64, which Metal does not have |
| W4 | Parameter accounting → N \approx 12 n_{\text{layer}}d^2; C \approx 6ND | A full GPT block, and a tiny model trained | MPS or CPU. SDPA on MPS refuses attention dropout |
| W5 | RoPE’s relative property in complex form; the phase argument for extrapolation | RoPE and its extrapolation variants | MPS; the long-context demo uses a quantized mid-size model |
| W6 | The linear-attention associativity identity → recurrent form; the SSD skeleton | Both algorithms for linear attention | MPS. SSM and hybrid models run on MLX’s own Metal kernels |
| W7 | Chinchilla’s first-order condition (one of the course’s three peaks) | A scaling law fitted at small scale | MPS — four or five tiny transformers |
| W8 | LoRA’s parameter and memory accounting; the SVD behind intruder dimensions | LoRA SFT with mlx_lm.lora |
MLX. Pointing it at a quantized model is QLoRA |
| W9 | DPO in four steps (one of the three peaks); GRPO’s advantage normalization | The DPO loss, with four tests | CPU |
| W10 | The two crossing pass@k curves; the binomial bound behind self-consistency | Budget forcing | MLX, a mid-size model with a thinking mode |
| W11 | Roofline and the batch threshold for decode; the three-line losslessness proof for speculative decoding | A roofline measured | MLX (mlx_lm.benchmark, --draft-model) |
| W12 | InfoNCE, and how the negatives define “similar”; RAG’s three-term decomposition | A minimal RAG, and three deliberate failures | CPU |
| W13 | The compound error rate p^n, and what a verifier does to it | An agent broken by prompt injection | A small local model is enough |
| W14 | The binomial standard error and paired bootstrap; the sparsity–reconstruction Pareto | An evaluation workshop (no new model) | CPU — TransformerLens avoids MPS by default |
Toolchain
Students need no compute for this course. It is lectures throughout, and every demo is run live in class on an Apple-silicon laptop with 64 GB of unified memory and no CUDA — which is what decides most of this table.
| Purpose | First choice | Notes |
|---|---|---|
| Inference for in-class demos | MLX (mlx-lm), or llama.cpp / Ollama (GGUF) |
Native to Apple silicon. Unified memory is the real advantage here: models far larger than 16 GB load comfortably |
| Building components by hand | Plain torch (MPS backend, or CPU) |
The hand-built labs in W3, W4, W6 and W9 are small enough that MPS — or even CPU — finishes them in minutes |
| Tokenizer experiments | tokenizers, sentencepiece, tiktoken |
Pure CPU, hardware-independent. The W2 fertility table |
| Fine-tuning demos | mlx_lm.lora |
W8. QLoRA is not a flag: point --model at an already-quantized model and that is QLoRA. Adapters are merged back with mlx_lm.fuse |
| Evaluation | mlx_lm.evaluate |
W14. The harness itself has no MLX backend — the integration lives on the MLX side and registers itself into lm-eval. The workshop looks at per-item results, not just aggregate scores |
| RAG | sentence-transformers + faiss-cpu, or plain numpy |
W12. Brute-force search on CPU is enough; no vector database needed |
| Interpretability | transformer-lens (runs on CPU in practice) |
Activation patching in W14. The library declines to auto-select MPS unless TRANSFORMERLENS_ALLOW_MPS=1, and currently marks no torch version as MPS-safe. CPU is fine at these model sizes |
What cannot be demonstrated on this hardware
| Not available | Why | What stands in for it |
|---|---|---|
bitsandbytes 4/8-bit, QLoRA |
CUDA only | mlx_lm.lora against a quantized model. But “you cannot do 4-bit in transformers on a Mac” stopped being true in 2026: transformers now ships a Metal quantization backend built on MLX’s kernels |
vLLM’s PagedAttention and continuous batching |
Not usable on Apple silicon | W11 teaches the argument and cites the SOSP 2023 numbers. PagedAttention’s memory argument was always clearest on a whiteboard |
| FP8 / NVFP4 measurements | Apple silicon has no such tensor formats | W7 and W11 cite the figures from the Quartet and NVFP4-pretraining reports |
| FlashAttention, custom Triton kernels | CUDA only | The IO argument in W3 is a board derivation, not an implementation exercise |
auto-gptq, autoawq |
CUDA | mlx-lm’s own AWQ / GPTQ / DWQ / dynamic-quant tools for the “what quantization changes besides perplexity” comparison |
float64 — finite-difference gradient checks, ill-conditioned linalg |
Metal has no double type. Not slow; absent | Pin those exercises to the CPU |
Attention dropout through scaled_dot_product_attention |
MPS’s SDPA refuses dropout_p > 0.0, and has no dedicated backward kernel |
The hand-built version writes its own dropout — which is one more argument for building it by hand |
Multi-process accelerate launch, DDP |
gloo and nccl do not work with mps |
Single device only. In-class demos never needed it |
CPU offload via device_map="auto" |
MPS requires the whole model to fit in unified memory | This is where 64 GB becomes a hard edge: too big is too big, with no automatic fallback |
Models used in the demos
The demos are not limited to models that fit a 16 GB card, so the sizes are chosen for what each point needs rather than for what is cheap to run:
| Used for | Suggested size | Why it is worth scaling up |
|---|---|---|
| Hand-built components (W3, W4, W6, W9) | Qwen3-0.6B / 1.7B | Small is the right choice here — the run finishes while the class is still watching |
| Long-context behaviour (W5) | A quantized mid-size model | The lost-in-the-middle curve is far more convincing on a real model than on 0.6B |
| MoE routing (W6) | A genuine open-weight MoE | No need to simulate with a tiny model plus four experts |
| Reasoning (W10) | A mid-size model with a thinking mode | Budget forcing and overthinking only show up on a real reasoning model |
| The 2026 flagships | Read the paper’s table | DeepSeek-V4, Kimi K3 and the rest are still out of reach — that has not changed |
Four Things This Course Admits It Does Not Know
These gaps are load-bearing rather than embarrassing — each is a project topic, and saying so is the honest graduate-level position.
- W2 — there is no verifiable published work on Traditional Chinese token efficiency. The fertility comparison has to be demonstrated live.
- W12 — chunking and query rewriting have no citable representative papers; both are engineering folklore more than literature, so the course substitutes an ablation lab. RAG evaluation has fragmented, and nothing has become the accepted successor to RAGAS.
- W13 — no textbook (JM3 ch 12 is unwritten), and no canonical design paper for the Model Context Protocol; its academic literature is almost entirely about security.
- W14 — item response theory for benchmarks is still scattered; the text-side benchmarks for Taiwanese Hokkien and Hakka are effectively absent; and model welfare has no citable academic literature.