自然語言處理與大型語言模型
Natural Language Processing and Large Language Models
The course is built around a single discipline: for every layer of a modern LLM, explain why it looks the way it does — and where it breaks.
| Lecturer | Hung-Shin Lee (李鴻欣) |
| Audience | Master’s / PhD students; no hard prerequisites |
| Format | 14 weeks × 3 hours, lectures throughout |
| Language | Lectures in Mandarin, slides in English |
| Compute | Nothing is required of students — every demo is run live in class |
| Project hardware | The term project assumes a single 16 GB CUDA GPU (RTX 5070 Ti class) |
| Assessment | Midterm report (40%), final paper (60%) |
| Requirements | Group work in teams of 2–4; two online discussion sessions |
Three Threads Through the Course
Every week comes back to these three questions. They are the coordinate system of the course, and the line that ties the 14 weeks together when you look back from the end.
| Thread | Question |
|---|---|
| Representation | What is the unit at this layer — byte, token, hidden state, KV entry, retrieved chunk, latent thought? Who chose that unit, and what does the choice cost downstream? |
| Compute | Does this method spend training compute or inference compute? Is it FLOPs bound or memory-bandwidth bound? |
| Supervision | Where does this capability come from — the pre-training distribution, human preference labels, a verifiable reward, retrieved knowledge, or extra compute at test time? The same behaviour can come from entirely different sources, and that is the central dispute in W10 and W14. |
Course Map
Endpoint — read a modern LLM layer by layer: for every layer from tokenizer to agent, say why it looks the way it does and where it breaks. W1 opens with the compression framing and the compute ledger that the remaining thirteen weeks keep filling in.
W3 and W4 are the hinge. Everything in Part III reads from two objects built there — the residual stream and the KV cache — so those two weeks are the ones not to miss.
Weekly Schedule
| Week | Topic | Part | Slides |
|---|---|---|---|
| W1 | From n-grams to Seq2Seq: A Compressed History 從 n-gram 到 seq2seq：一條被壓縮的技術史 |
Architecture | ▶ Open |
| W2 | Tokenization: The World the Model Sees Tokenization：模型看到的世界，以及它看不到的 |
Architecture | — |
| W3 | Transformer I: Attention Mechanics, Complexity, and Building It by Hand Transformer I：attention 的機制、複雜度與手刻 |
Architecture | — |
| W4 | Transformer II: The Full Block, Parameter Accounting, and Training Dynamics Transformer II：完整 block、參數記帳與訓練動力學 |
Architecture | — |
| W5 | Positional Encoding, Long Context, and the Limits of Context 位置編碼、長文本與 context 的極限 |
Architecture | — |
| W6 | Architecture Routes: MoE, SSMs, and Linear Attention 架構路線：MoE、SSM 與 linear attention |
Architecture | — |
| W7 | Pre-training: Scaling Laws and Data Curation Pre-training：scaling laws 與資料策展 |
Training | — |
| W8 | Post-training I: SFT, PEFT, and Forgetting Post-training I：SFT、PEFT 與遺忘 |
Training | — |
| W9 | Post-training II: Preference Alignment and Reinforcement Learning Post-training II：偏好對齊與強化學習 |
Training | — |
| W10 | Reasoning and Test-Time Compute Reasoning 與 test-time compute |
Training | — |
| W11 | Inference Efficiency: From FLOPs to Memory Bandwidth 推論效率：從 FLOPs 到 memory bandwidth |
Systems & Scrutiny | — |
| W12 | RAG and Context Engineering RAG 與 context engineering |
Systems & Scrutiny | — |
| W13 | Agentic Systems: Tools, Planning, Failure Modes, and Security Agentic systems：工具、規劃、失效與安全 |
Systems & Scrutiny | — |
| W14 | Evaluation, Interpretability, Safety, and Open Problems 評估、可解釋性、安全，與 open problems |
Systems & Scrutiny | — |