W1｜A Systems View of Spoken Dialogue and the Latency Budget

Part I — Foundations · 語音對話系統的系統觀與延遲預算

▶ Slides for this week

Where This Fits

The course does not open with signal processing; it opens with the endpoint. Week 1 lays out the system the remaining thirteen weeks are built towards, so that every module introduced later has a slot to go back into, and it establishes the habit of treating latency as a first-class design constraint rather than as something to tune once everything else works.

Two items that belong to no single module close the week. The first is data: Taiwanese Hokkien, Hakka and the Formosan languages have neither public two-channel conversational corpora nor a settled orthography — a structural research gap, not an engineering chore. The second is a set of four questions to put to any paper in this field — is the input the same, how was the timing measured, what was normalised away, and what is this metric actually rewarding — which come back every week for the rest of the course.

Learning Objectives

  1. Draw the data flow of three architectures — cascaded ASR→LLM→TTS, end-to-end speech-to-speech, and native full-duplex — and identify the information bottleneck in each.
  2. Decompose a target end-to-end latency (500 ms, say) into a per-module budget, say which items are algorithmic latency and which are computational latency, and state what each cell actually measures: the time that elapses after the user stops speaking, not the module’s total processing time.
  3. Define the operational difference between half-duplex and full-duplex speech interaction, and name four interactive behaviours a half-duplex system cannot produce: backchannels, barge-in, holding the floor through overlap, and proactive turns.
  4. Use the mismatch between human transition timing and the time it takes to plan an utterance to argue that turn-taking is a prediction problem rather than a detection problem — and therefore that making the VAD faster is not the same as solving it.
  5. Put four diagnostic questions to any paper in this field — is the input the same, how was the timing measured, what was normalised away, and what is the metric rewarding — and say which of the four that paper is weakest on.

Demos

Of this week’s two demos, one is a program; it is published so that you can read it, re-run it, and change it — you are not required to run anything, and the numbers on the slides are the rehearsal records checked in with the code. Each folder’s README.md gives requirements, setup, commands, and what you should see; the index at demos/ explains the shared environment.

Slides What it shows Needs Runs on
Demo 1
cold open (pre-recorded)		 p2–p5, p45 Why: to hear the two opposite failures of a spoken dialogue system before any definitions — speaking when it should listen, and staying silent when it should speak.
What you hear: two recordings from Kyutai’s interactivity-alignment-samples (CC BY 4.0): the same pre-recorded user, the same model family, only the post-training differs (left channel = user, right = model). No program to run: the audio is in slides/assets/w01/ (*-base / *-fisher) and the two scripts that measured the overlap and drew the timelines are in slides/scripts/.
Result: one talks over the user for 2.90 s and then says nothing for 10.2 s; the other overlaps for 0.00 s and answers 60 ms after the user finishes		 — any player
Demo 2
one sentence through a cascade, budget filled in live		 p41–p42 Why: the latency budget, filled in from a real pipeline: which cell is largest, and is it compute or a setting?
What you see: one recorded sentence with a pause in it → energy-VAD endpoint → whisper.cpp → Gemma 4 26B-A4B via mlx-lm → Kokoro via mlx-audio, every boundary time-stamped, ten runs, p50 / p95; the endpoint subcommand shows what 300 / 700 / 1000 / 1500 ms thresholds do to the pause inside the sentence.
Result: endpoint threshold 1000 ms, ASR tail 70, LLM first token 94, TTS first packet 124, total 1388 ms — the largest cell is the silence threshold, not the biggest model		 pywhispercpp + mlx + mlx-lm + mlx-audio, ≈ 15 GB of models; the recording is the lecturer’s and is not published — record your own Apple silicon only, 32 GB

Tested on one machine only — an Apple M5 Max (64 GB, no CUDA), macOS 26.6, Python 3.12 — with the versions pinned in demos/versions.lock. Terminal prompts are in Chinese; Enter advances, q quits.

References

  • JM3 Ch 26 Conversation and its Structure (the linguistic account of conversational structure and turn-taking); App K Frame-based Dialogue Systems (the classical pipeline, as a point of comparison)
  • Sacks, Schegloff & Jefferson, A Simplest Systematics for the Organization of Turn-Taking for Conversation, Language 50(4), 1974 — the paper JM3 Ch 26 is built on. At graduate level, read the original rather than the textbook summary
  • Stivers et al., Universals and Cultural Variation in Turn-Taking in Conversation, PNAS 106(26): 10587–10592 (2009, open access) — ten languages on five continents, and the evidence behind the minimal gap, minimal overlap norm that the whole latency budget rests on. Read the numbers off the figures yourself rather than quoting the single mean that circulates second-hand
  • Ohashi, Zeghidour, Défossez & Kharitonov, Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models, EMNLP 2026 (arXiv:2606.11167) — the source of the audio used in the week 1 cold open (kyutai/interactivity-alignment-samples, CC-BY-4.0). We listen to it in week 1 and read it in W14; the loop is deliberate
  • Ji et al., WavChat: A Survey of Spoken Dialogue Models, arXiv:2411.13577 (2024) — a taxonomy of half-duplex-era systems; the right level of overview for week 1
  • Lu et al., A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machine, arXiv:2606.19453 (2026) — the most recent and most complete taxonomy of full-duplex architectures, and the backbone of W12–W13
  • Chen & Yu, From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models, arXiv:2509.14515 (2025)
  • Wang et al., Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking, arXiv:2511.01299 (2025) — listening and speaking under a single framework
  • Timing statistics for human turn-taking (further searching): turn-taking gap distribution, response latency conversation corpus, floor transfer offset
回到頂端