Syllabus

Setup, course structure, the compute ledger, and how to read a paper

Course Setup

Item	Setting
Course	自然語言處理與大型語言模型 · Natural Language Processing and Large Language Models
Lecturer	Hung-Shin Lee (李鴻欣)
Audience	Master’s / PhD students
Prerequisites	None required; linear algebra, probability and Python are soft expectations
Format	14 weeks × 3 hours, lectures throughout
Language	Lectures in Mandarin, slides in English
Compute	Nothing is required of students. The course is lectures throughout, and every demo is run live in class on the lecturer’s machine
Project hardware	The term project assumes a single 16 GB CUDA GPU (RTX 5070 Ti class) — the Supplements topics were selected against that budget
Assessment	Midterm report (40%), final paper (60%) — see Supplements for what each one has to contain
Depth	Graduate — derivations, the literature behind each idea, open problems
Requirements	Group work in teams of 2–4; two online discussion sessions

註釋On “no prerequisites” together with “full derivations”

The two are in tension, and the course accepts that deliberately. The catch-up load sits in W1 and the first half of W3, and week 1 hands out a two-page list of the mathematics assumed: inner products, projection and the intuition for an SVD; conditional independence, expectation and variance; gradient descent and the chain rule. W7 (Chinchilla’s first-order condition), W9 (the DPO derivation) and W3–W4 (the variance argument and parameter accounting) are the three mathematical peaks, each needing 40–60 minutes at the board. W9 is the most dangerous week for a student with no RL background, so it carries a second explanation — preference optimization as weighted maximum likelihood — for exactly that reason.

Course Structure

Part	Weeks	Content
Part I — Architecture	W1–W6	A compressed history from n-grams to seq2seq; tokenization; attention’s mechanics, built by hand; the full block and training dynamics; positional encoding and long context; MoE, state-space models and linear attention
Part II — Training	W7–W10	Scaling laws and data curation; SFT, PEFT and forgetting; preference alignment and RL; reasoning and test-time compute
Part III — Systems & Scrutiny	W11–W14	Inference efficiency, from FLOPs to memory bandwidth; RAG and context engineering; agentic systems and their failure modes; evaluation, interpretability and safety

W3 and W4 are the hinge of the course — W5, W6 and W11 rest entirely on understanding the residual stream and the KV cache, so neither is a week to miss. W2 is a prerequisite for W7, because the tokenizer shifts the compute-optimal configuration; W9 is a prerequisite for W10.

Two metaphors are meant to run the length of the course, and each is introduced with an explanation of where it fails. Compression arrives in W1 and returns in W7 (a scaling law is the regularity in the compression ratio), W11 (quantization compresses the compressor) and W12 (RAG declines to compress at all). The residual stream as a shared bus arrives in W4 and returns in W6 (swapping out how data is moved), W8 (a low-rank bypass on the bus) and W14 (a circuit is a signal traced along it).

The Compute Ledger: The Course’s Shared Coordinate System

You get this in week 1, and each week adds the layer it is responsible for. It is what lets an abstract argument fall back onto a number at any point.

Where the cost falls	Formula	What actually binds it
Training — one run	C_{\text{train}} \approx 6ND FLOPs	Compute. The 6 is 2 forward +\ 4 backward, per parameter per token
Prefill — once per request	C_{\text{pre}} \approx 2NL_{\text{in}} FLOPs	Compute. The whole prompt goes through together, so the matrices are large enough to keep the GPU busy
Decode — per generated token	C_{\text{dec}} \approx 2N FLOPs	Memory bandwidth. One token’s worth of arithmetic against a full read of all N weights out of HBM
KV cache — state carried, not FLOPs	M_{\text{KV}} = 2\,n_{\text{layer}}\,n_{\text{kv}}\,d_{\text{head}}\,L\,b bytes	Capacity first, then bandwidth. The leading 2 is key + value; the whole thing is re-read at every decode step
MoE — the two columns come apart	memory \propto N_{\text{total}}, FLOPs \propto N_{\text{active}}	Both, but separately — which is the entire point of the architecture (W6)

N parameters · D training tokens · L_{\text{in}} prompt length · L context length so far · b bytes per element (2 at FP16) · n_{\text{kv}} key/value heads (=n_{\text{head}} for MHA, 1 for MQA).

The line worth memorising. The first two rows are arithmetic problems; the last two are traffic problems. Decode is not short of FLOPs — it is short of bandwidth, and the KV cache is what fills the road. Most of Part III is an attempt to make those two rows cheaper.

Two numbers to work out for yourself in the first session, and to revisit as the course goes:

1. The training FLOPs for a 7B dense model at Chinchilla-optimal (D \approx 20N), and how many years that is on a single consumer GPU — an RTX 5070 Ti, say, which is the compute a term project actually has.
2. The KV cache for that model at batch 1, 32k context, FP16 — in GB, against two real ceilings: the 16 GB on that same card, which cannot hold it, and the 64 GB of unified memory the classroom demos run on, which can, at a fraction of a datacenter GPU’s bandwidth. Capacity and bandwidth are two separate limits, and 64 GB relieves only the first — which is the first hint of the roofline in W11. That number is the shared motivation for W5, W6 and W11.

A third number, working backwards from the MoE activation ratio to why a 2026 flagship is rational at roughly a trillion total parameters and tens of billions active, opens W6 instead — it needs the idea of an active parameter, which arrives that week.

Four Questions for Reading a Paper

Handed out in week 1 and used every week after. They are short on purpose — they have to be usable in the ten minutes before a seminar.

1. Was the baseline actually tuned? The most common source of illusory progress in this field. W8 (LoRA variants) and W9 (GRPO variants) both have concrete cases where the improvement disappears once the baseline gets a learning-rate sweep.
2. Is the comparison at equal compute? Iso-FLOPs, iso-parameter and iso-latency are three different comparisons, and they frequently give opposite answers. W6 is built around this.
3. What is the metric rewarding? The gap between a metric and the capability it stands for is the subject of W14, but it comes up every week — most sharply in W11, where KV-cache compression leaves perplexity almost untouched and quietly breaks instruction following.
4. Is the claim mechanistic or correlational? “The attention weight is high”, “the chain of thought wrote this step”, “the reward went up” are all correlational. W3, W10 and W14 are the same error three times over, and the answer is the same each time: ask for an intervention, and ask what the strong baseline does.

One Argument, Made Three Times

Question 4 above is the abstract version. Here is the concrete one: the course meets the same mistake in three different places, and naming it is what the last week is for.

Week	A readable intermediate artifact	Misread as
W3	attention weight	importance, or an explanation
W10	chain of thought	the model’s reasoning process
W14	sparse-autoencoder feature	a concept inside the model

Each time the answer is the same: ask for an intervention rather than a correlation, and ask what a strong baseline does on the same measurement.