語音處理與人機互動
Speech Processing and Human-Machine Interaction
The course is built backwards from a single question: how do you build a spoken dialogue system that can listen while it speaks, be interrupted, and know when it is its turn to talk?
| Lecturer | Hung-Shin Lee (李鴻欣) |
| Audience | Master’s / PhD students; no hard prerequisites |
| Format | 14 weeks × 3 hours, lectures throughout |
| Language | Lectures in Mandarin, slides in English |
| Compute | Nothing is required of students — every demo is run live in class |
| Project hardware | The term project is planned around a single 16 GB CUDA GPU (RTX 5070 Ti class), which you provide yourself |
| Assessment | Midterm report (40%), final paper (60%) |
| Requirements | Group work in teams of 2–4; two online discussion sessions |
Three Threads Through the Course
Every week comes back to these three questions. They are the coordinate system of the course, and the line that ties the 14 weeks together when you look back from the end.
| Thread | Question |
|---|---|
| Representation | What does this layer operate on — waveform / spectrogram / continuous SSL feature / discrete token? And what decides the frame rate? |
| Latency | How much of the latency budget does this module spend? Is it algorithmic latency (lookahead) or compute latency? |
| Supervision | Where does this capability come from — labeled data / self-supervision / synthetic data / human preference (RLHF)? |
Course Map
Endpoint — a full-duplex spoken dialogue system: listen while speaking, survive barge-in, and answer within roughly 300 ms. The 14 weeks are built backwards from it; W1 sets out the systems view and the latency budget that the rest has to fit.
W1Systems view & latency W2Signals & front-end W3Alignment: HMM → CTC W4Sequence models & streaming W5SSL representations W6Codecs & discretization
W6 is the hinge. Everything in Part III rests on what is decided there — the frame rate, the token budget, and the split between semantic and acoustic information — so that is the week not to miss.
Weekly Schedule
| Week | Topic | Part | Slides |
|---|---|---|---|
| W1 | A Systems View of Spoken Dialogue and the Latency Budget 語音對話系統的系統觀與延遲預算 |
Foundations | ▶ Open |
| W2 | Speech Signals, Auditory Front-Ends, and the Physics of Representation 語音訊號、聽覺前端與表徵的物理基礎 |
Foundations | ▶ Open |
| W3 | The Alignment Problem: From HMM to CTC 對齊問題：從 HMM 到 CTC |
Foundations | ▶ Open |
| W4 | Sequence Models and Streaming Architectures: Transformer, Conformer, RNN-T 序列模型與 streaming 架構：Transformer、Conformer、RNN-T |
Foundations | ▶ Open |
| W5 | Self-Supervised Representation Learning: Where Speech Foundation Models Come From 自監督表徵學習（SSL）：語音基礎模型的來源 |
Foundations | ▶ Open |
| W6 | Neural Audio Codecs and Discretization: Turning Speech into a Language Neural Audio Codec 與離散化：語音成為「語言」的關鍵一步 |
Foundations | ▶ Open |
| W7 | Speech Recognition: Three Paradigms, Streaming, and the LLM Turn 語音辨識：三種範式、streaming 與 LLM 化 |
Modules | — |
| W8 | Speech Synthesis I: Generative Paradigms and Streaming TTS 語音合成 I：生成範式與 streaming 合成 |
Modules | — |
| W9 | Speech Synthesis II: Controllability, Post-Training Alignment, Evaluation, and Misuse 語音合成 II：可控性、後訓練對齊、評估與濫用防範 |
Modules | — |
| W10 | The Physical Layer of Full-Duplex: VAD, AEC, Enhancement, Separation, Speaker 全雙工的「物理層」：VAD、AEC、增強、分離與說話人 |
Modules | — |
| W11 | Audio-Native Language Models: Building Half-Duplex Spoken Dialogue Audio-Native 語言模型：半雙工語音對話的建成 |
Dialogue Systems | — |
| W12 | Full-Duplex I: Linguistic Foundations and Modeling of Turn-Taking 全雙工 I：Turn-Taking 的語言學基礎與建模 |
Dialogue Systems | — |
| W13 | Full-Duplex II: Architectures, Multi-Stream Modeling, and Data 全雙工 II：架構路線、多串流建模與資料問題 |
Dialogue Systems | — |
| W14 | Evaluation, Alignment, Deployment, and Open Problems 評估、對齊、落地與 Open Problems |
Dialogue Systems | — |