Syllabus

Course setup, structure, and the latency budget

Course Setup

Item	Setting
Course	語音處理與人機互動 · Speech Processing and Human-Machine Interaction
Lecturer	Hung-Shin Lee (李鴻欣)
Audience	Master’s / PhD students
Prerequisites	None required; linear algebra, probability and Python are soft expectations
Format	14 weeks × 3 hours, lectures throughout
Language	Lectures in Mandarin, slides in English
Compute	Nothing is required of students in class. The course is lectures throughout, and every demo is run live on the lecturer’s machine
Project hardware	The term project is planned around a single 16 GB CUDA GPU (RTX 5070 Ti class) — a planning baseline, not a cap — which you provide yourself, usually your lab’s machine; the problem cards in Supplements show how to budget memory against it
Assessment	Midterm report (40%), final paper (60%) — see Supplements for what each one has to contain
Depth	Graduate — derivations, the literature behind each idea, open problems
Requirements	Group work in teams of 2–4; two online discussion sessions

On “no prerequisites” together with “full derivations”

The two are in tension, and the course accepts that deliberately. The catch-up load is concentrated in the second half of W2 (signals and representation) and the first half of W3 (a fast review of HMMs), and week 1 hands out a two-page list of the mathematics assumed: inner products and projection from linear algebra, conditional independence and marginalization from probability, and gradient descent. W3, W6 and W8 are the three mathematical peaks; each needs an hour or more at the board.

Course Structure

Part	Weeks	Content
Part I — Foundations	W1–W6	Systems view and the latency budget; signals and the front end; alignment (HMM → CTC); Transformers and streaming; SSL representation learning; neural codecs and discretization
Part II — Modules	W7–W10	ASR; TTS generative paradigms; TTS controllability and evaluation; the acoustic front end (VAD / AEC / enhancement / separation / speaker)
Part III — Dialogue Systems	W11–W14	Audio-native LMs (half-duplex); modeling turn-taking; full-duplex architectures and data; evaluation and deployment

W6 — neural codecs and tokenization — is the hinge of the course: W11–W14 rest entirely on frame rate, token budget, and the separation of semantic from acoustic information.

The Latency Budget: The Course’s Shared Coordinate System

Hand this diagram out in week 1, then fill in each module’s contribution as the course goes.

Human turn transitions are a distribution, not a single mean. Across ten languages on five continents, informal conversation follows one norm — minimal gap, minimal overlap — with transitions clustering near zero and cross-language means differing by at most about ±250 ms. Responses that do not answer the question, or that run against its bias, are delayed by up to a second, and listeners read that delay as meaning (Stivers et al., PNAS 106(26): 10587–10592, 2009). The engineering target for “natural” comes from that distribution, not from the single figure that usually circulates second-hand.

Each cell holds the time that elapses after the user stops speaking — not the module’s total processing time. A streaming ASR has already consumed most of the utterance by then; counting that work again inflates the whole budget by an order of magnitude.

The template is cascade-shaped. A native full-duplex model has no endpoint-detection cell and no ASR tail: it emits on every frame. The budget stops fitting in W13, and that is the definition of full duplex rather than a flaw in the table.

user stops speaking
      ──▶ [endpoint detection] ──▶ [ASR tail]
      ──▶ [LLM prefill + first token]
      ──▶ [TTS first packet] ──▶ [playout buffer]
      ──▶ user hears

回到頂端