Materials and Resources
Textbook chapters, one paper per week, toolchain
Core Textbooks
| Code | Source | Used for |
|---|---|---|
| JM3 | Jurafsky & Martin, Speech and Language Processing, 3rd ed. online draft, 2026-08-19 release (https://web.stanford.edu/~jurafsky/slp3/) | The teaching baseline for the speech and dialogue chapters; official slides available to adapt |
| HAH | Huang, Acero & Hon, Spoken Language Processing, Prentice Hall, 2001 | Solid derivations for signal processing, hearing and HMMs (old, but nothing has replaced it) |
| YD | Yu & Deng, Automatic Speech Recognition: A Deep Learning Approach, Springer, 2015 | Hybrid DNN-HMM through sequence discriminative training |
| TAN | Xu Tan, Neural Text-to-Speech Synthesis, Springer, 2023 | Systematic treatment of TTS (check chapter numbers against the printed book) |
| LOI | Loizou, Speech Enhancement: Theory and Practice, 2nd ed., CRC Press, 2013 | Statistical methods and evaluation metrics for enhancement |
| VVG | Vincent, Virtanen & Gannot, Audio Source Separation and Speech Enhancement, Wiley, 2018 | Separation and multichannel processing |
| BB | Bishop & Bishop, Deep Learning: Foundations and Concepts, Springer, 2024 | The mathematics under diffusion, normalizing flows and VAEs |
⚠️ JM3 chapter numbers move between draft releases. This course fixes the numbering of the 2026-08-19 release: Ch 15 Phonetics and Speech Feature Extraction, Ch 16 Automatic Speech Recognition, Ch 17 Text-to-Speech, Ch 26 Conversation and its Structure, App A Hidden Markov Models, App K Frame-based Dialogue Systems. Cite one release consistently when building slides.
One Paper per Week
| Week | Topic | If you read only one |
|---|---|---|
| W1 | Systems view | Lu et al., A Survey of Full-Duplex Spoken Dialogue Systems, arXiv:2606.19453 |
| W2 | Signals and front end | JM3 Ch 15 |
| W3 | Alignment | Graves et al., CTC, ICML 2006 |
| W4 | Sequence models and streaming | Graves, Sequence Transduction with RNNs, arXiv:1211.3711 |
| W5 | SSL | Hsu et al., HuBERT, TASLP 2021 |
| W6 | Codec / tokenization | Défossez et al., Moshi (the Mimi codec section) |
| W7 | ASR | Radford et al., Whisper, ICML 2023 |
| W8 | TTS I | Lipman et al., Flow Matching for Generative Modeling, ICLR 2023 |
| W9 | TTS II | Rafailov et al., DPO, NeurIPS 2023 + ITU-T P.808 |
| W10 | Acoustic front end | Wang & Chen, Supervised Speech Separation: An Overview, TASLP 2018 |
| W11 | Audio-native LM | Défossez et al., Moshi, arXiv:2410.00037 |
| W12 | Turn-taking | Skantze, Turn-Taking in Conversational Systems: A Review, CSL 2021 |
| W13 | Full-duplex architectures | Nguyen et al., dGSLM, TACL 2023 |
| W14 | Evaluation and alignment | Lin et al., Full-Duplex-Bench, ASRU 2025 |
Toolchain
You need no compute for this course. It is lectures throughout, and every demo is run live in class on an Apple-silicon laptop with 64 GB of unified memory and no CUDA. That last constraint is what decides most of this table — and it decides it more sharply for speech than it would for a text-only course, because a good deal of the classical ASR training stack is written against CUDA kernels.
| Purpose | First choice | Notes |
|---|---|---|
| ASR, most mature path | whisper.cpp |
Apple silicon is a first-class target: inference runs on the GPU through Metal, and Core ML can put the encoder on the Neural Engine. The first run after install is slow while Core ML compiles |
| The MLX speech stack | mlx-audio |
The actively developed one — text-to-speech, speech-to-text, separation, enhancement and VAD in a single package |
| NVIDIA’s ASR models, on a Mac | parakeet-mlx |
Parakeet with a streaming API. Worth noting that the models run here even though the framework they came from does not |
| Signal processing and metrics | librosa, torchaudio, jiwer, pesq, pystoi |
Hardware-independent. Note that torchaudio is in maintenance mode: load() and save() are now TorchCodec, and sox_effects is gone |
| Speech modules (VAD, separation, speaker) | SpeechBrain, pyannote.audio, Silero VAD | Silero is fast enough on CPU that the question does not arise. pyannote 4 needs ffmpeg installed |
| Codec | Mimi | EnCodec (2022) and Descript Audio Codec (2023) are both frozen; Mimi is the one still moving, and it is what the full-duplex models in W13 use |
| TTS | F5-TTS, or the TTS models in mlx-audio |
Weight download is the real delay — warm it up before class |
| Full duplex | moshi-mlx |
This is the reason W12 and W13 can be shown live rather than played from a recording. The PyTorch build of the same model needs a 24 GB GPU and has no quantization, so on this machine MLX is not the fallback — it is the only path |
What cannot be shown on this hardware
| Not available | Why |
|---|---|
bitsandbytes, vLLM, FlashAttention, custom Triton kernels |
CUDA only |
| NeMo | Built on NVIDIA’s stack; its requirements begin with an NVIDIA GPU |
GPU inference in faster-whisper |
It is built on CTranslate2, whose GPU backends are CUDA and ROCm only. It runs here, but on the CPU — the “Apple Accelerate” it advertises is a CPU math library, not GPU acceleration. That distinction is worth holding on to: “Apple silicon acceleration” means at least four different things — Metal, the Neural Engine through Core ML, PyTorch’s MPS backend, and Accelerate — and a tool that supports one of them supports none of the others by default |
| CosyVoice | No macOS or Apple-silicon support in its documentation |
The limitation is part of the material. Saying in class that a demo cannot run here, and naming which kernel is missing, makes the point better than a slide can: in speech, what a system can do is bound to what it runs on. That is the latency axis, stated concretely.
Code
Everything public lives in one repository: github.com/hungshinlee/Speech-AI (the GitHub icon in the top-right corner goes there too).
- Slides —
slides/wNN.qmd, rendered at Slides. They are Quarto reveal.js decks; the source is readable as Markdown. - Analysis scripts referenced on the slides —
slides/scripts/, e.g.analyze_duplex_audio.py(energy VAD, overlap / response / stop latency) andmake_duplex_timeline.py(two-track timeline). Standard library +numpy/scipy/matplotlib; run them on any stereo WAV with the user on the left channel and the system on the right. - Weekly pages are generated from the course outline; the outline itself is not published.
Practical Notes on the Demos
The demos are run live in class by the lecturer on the machine described above; you do not need to set anything up. Three things about them are worth knowing:
- The first weight download and the first Metal / Core ML compile often take longer than the inference itself, so every model is loaded and warmed up before class. If a demo is played back from a rehearsal recording instead of run live, the screen says so.
- A few dozen utterances from LibriSpeech
dev-cleanplus MUSAN are enough for everything shown; nothing needs a large dataset. - The six “play the broken one first” demos are the most memorable part of the course: Griffin-Lim phase reconstruction (W2), the CTC blank posterior heatmap (W3), Whisper hallucination (W7), vocoder artifacts (W8), WER going up after enhancement (W10), and barging in versus responding too slowly (W12).