W6｜Neural Audio Codecs and Discretization: Turning Speech into a Language

Part I — Foundations · Neural Audio Codec 與離散化：語音成為「語言」的關鍵一步

▶ Slides for this week

Where This Fits

The hinge of the course. W5’s units were predicted — the target was “what is the masked frame”; this week’s tokens are reconstructed — the target is the waveform itself. Three things change once the target is the waveform. The quantiser has to carry timbre and phase, not just content, so one codebook is not enough and codebooks are stacked into a residual VQ. There is no contrastive task to help, so codebook collapse returns in a harsher form and with more remedies (EMA, k-means initialisation, dead-code reset, low-dimensional codebooks, FSQ). And the rate of tokens becomes a number somebody has to design for the first time — it sets the context an LM consumes, the number of decoding steps per second, and the latency floor at once. This week decides whether you can read W11–W13: every audio-native language model there is a language model over this week’s tokens, and their architectural differences (multi-stream, RQ-Transformer, delay patterns) are mostly different answers to the fact that one time step carries N_q tokens.

The main thread is that a codec has three design freedoms that constrain each other: (1) the quantiser — VQ / RVQ / FSQ, how many codebooks, how large; (2) the training objective — a reconstruction loss alone gives over-smoothed output (W2 listed this among the three reasons vocoders exist), so discriminators and feature matching are added, and what is reconstructed decides what the tokens contain: purely reconstruction-driven tokens entangle content, timbre and environment, which is why SpeechTokenizer and Mimi distil an SSL representation into the first codebook — W5’s units brought back inside the codec; (3) the frame rate — 75 Hz and 12.5 Hz differ by a factor of six, and a low frame rate is kind to the LM but not to quality or to the granularity of latency. The second thread is two kinds of token: acoustic tokens (SoundStream / EnCodec / DAC, reconstruction-driven) and semantic tokens (W5’s k-means units, prediction-driven); AudioLM was the first model to stack the two, and most speech tokenisers since 2023 ask whether one tokeniser can do both. The third thread is arithmetic: bitrate, token rate, context length, decoding steps, latency floor — each a function of the three freedoms, and each worked out in class.

Where the three course-long axes sit this week. Representation: the rightmost of W2’s trio (log-mel / SSL feature / discrete token) is filled in, and for the first time “who decides the frame rate” is answered by a deliberate choice: in W2 it was the hop, in W4 / W5 the stride of a CNN; this week Mimi fixes its frame rate at 12.5 Hz (80 ms per frame, Moshi §3.3.1) for the sake of the LM’s sequence length rather than the signal. Latency: a codec’s algorithmic delay is at least one frame (80 ms at 12.5 Hz) plus the encoder’s lookahead; a causal codec drives the lookahead to zero at a price in quality; and since W13’s full-duplex models make one decision per frame, the frame rate is also the granularity of reaction — this is the cell of W1’s latency budget filled in this week. Supervision: a codec needs no labels, but the target changes from “built by the model itself” (W5) to “the waveform plus an adversary”; semantic distillation then uses W5’s trained SSL model as a teacher — the second time the answer to “where does the supervision come from” changes.

The local-language thread takes, for the first time, a shape with peer-reviewed evidence: what W2 forecast and W5 inferred — “semantic tokens flatten tone” — Osakuade & King (Speech Prosody 2026) measured on Mandarin and Yorùbá: the continuous SSL representations encode tone, but once quantised into discrete units tone is lost more readily than segmental information. That is evidence at the level of SSL units; whether acoustic codecs keep tone had, as of 2026-10-16, no peer-reviewed study we could find (the probe of Mimi’s first codebook looks at English segments only — and finds that segmental information is not confined to the first codebook, whose tokens correspond to allophones rather than phonemes). So this week’s position is: Mimi and SpeechTokenizer distil WavLM / HuBERT into the first codebook, which distils “tone is not in the units” along with it, leaving tone to the acoustic layers — a prediction from the design, to be listened to in Demo 1 on a Mandarin sentence; the search pointer stays.

Learning Objectives

1. Derive the three terms of the VQ-VAE loss (reconstruction, codebook, commitment) and the straight-through gradient, say where straight-through is biased and why the commitment loss bounds that bias, and write down the EMA codebook update and show that it is online k-means.
2. Write the residual-VQ recursion and the bitrate formula B = f_r N_q \log_2 V, explain from the decay of residual energy across stages why the first stages have the largest marginal gain and why quantizer dropout lets one model serve several bitrates, and distinguish “the first codebook carries most of the energy” from “the first codebook carries the semantics” — the latter holds only through distillation.
3. Tell apart the causes of codebook collapse (a codebook initialised away from the data; straight-through updating only the selected code) and five remedies (EMA, k-means initialisation, dead-code reset, low-dimensional normalised lookups, FSQ), and explain why FSQ has no collapse by construction and what it costs.
4. Compare acoustic tokens, semantic tokens, and the three designs that try to unify them (SpeechTokenizer’s distillation, Mimi’s split RVQ, single-codebook tokenisers) against two goals — feeding an LM and reconstruction quality — and explain why AudioLM had to stack the two kinds of token.
5. For a given codec configuration, compute the token rate, the context consumed by a 30-minute conversation, the decoding steps per second (flattened vs. RQ-Transformer), and the latency floor, and fill in the corresponding cell of W1’s latency budget.

Demos

The demos are programs and are published so that you can read them, re-run them, and change them — you are not required to run anything. Each folder’s README.md gives requirements, setup, commands, and what you should see; the index at demos/ explains the shared environment.

	Slides	What it shows	Needs	Runs on
Demo 1 adding the stages back, one at a time: Mimi, EnCodec, DAC	p29–p31	Why: every claim about what the residual-VQ stages carry — energy first, detail later, content only through distillation — can be heard: reconstruct the same sentence with more and more stages, and with stage 1 alone. What you see: the W3 LibriSpeech sentence and one Mandarin sentence through Mimi (n_q = 1, 2, 4, 8), EnCodec 24 kHz (1.5–24 kbps) and DAC 24 kHz (n_q = 1–8), each codec encoded once and decoded from the first n_q rows of its code matrix; the screen shows f_r, n_q, V, B, the reconstruction’s log-mel and, on request, the F_0 tracks. Result: the lecture predicted that Mimi’s stage 1 keeps the words and loses the voice, that EnCodec at ten times the bitrate does the opposite, and that the four Mandarin tones are flattened at stage 1. In rehearsal (one listener, two sentences) the first prediction failed — stage 1 alone (138 bps) kept neither the words nor the voice, and EnCodec at 1.5 kbps and DAC at 750 bps were clearer on both counts — while the tone prediction held: the Mandarin tones were flattened at stage 1 (F_0 RMSE 57 Hz, voicing agreement 0.49) and back at stage 8 (5.4 Hz, 0.86) and in EnCodec at 1.5 kbps (23 Hz, 0.86); every objective number is in runs/rehearsal/	torch 2.14.0 + transformers 5.17.0 (no new packages); three models, a few hundred MB; the LibriSpeech sentence is in the folder, the tonal sentence you record yourself	codecs on mps or CPU (a sentence takes seconds either way); metrics on CPU
Demo 2 what the letters look like: token matrix, codebook usage, residual energy, the bill	p32–p33, p42	Why: three things on the slides are drawn until now — the residual curve, “stage 1 is the most language-like”, and how many codes each stage uses — and all three can be measured on real codecs; then the arithmetic of the bill. What you see: the W3 sentence’s N_q \times T matrix through Mimi, EnCodec and DAC (runs of repeated tokens visible as colour); on the W5 dev-clean subset (40 speakers × 10 sentences) per stage the codes used over V, H_1 / \log_2 V, the repeat rate, the residual-energy curve \sum_t \lVert \mathbf{r}_{t,i} \rVert^2 / \sum_t \lVert \mathbf{z}_t \rVert^2 with Mimi’s semantic and acoustic paths separate, and H_2 / H_1 with a shuffled baseline; then token rate, 30-minute context, steps per second and 1/f_r for three settings beside the same 30 minutes as text tokens. Result (rehearsed 2026-10-18; all three weights have 32 stages): stage 1 does have the longest runs (repeat rate EnCodec 0.19 vs. 0.04 at stage 8, DAC 0.07 vs. 0.00; Mimi 0.06 at 80 ms per frame) and, for EnCodec and DAC, the most skewed usage (H_1/\log_2 V 0.78 vs. 0.85–1.00 later); Mimi’s semantic VQ is not more skewed than its acoustic stage 1 (0.90 vs. 0.89). EnCodec’s late stages have dead codes (507 of 1024 used at stage 32); DAC uses all 1024 from stage 2 on. Residual energy: EnCodec 0.034 after stage 1 and 0.001 at 32; Mimi acoustic 0.17 → 0.009, semantic VQ alone 0.36 — the distilled stage carries less energy than the reconstruction-trained one; DAC is not monotone (0.67 → 0.40 at stage 9 → 0.72 at 32), because its nearest-neighbour search runs in an 8-d projected space, not in the 1024-d residual space. Temporal structure H_2/H_1: Mimi semantic 0.36 (acoustic 0.46), EnCodec stage 1 0.67 (later 0.77), DAC 0.80 — and DAC’s later stages equal their shuffled baseline exactly. The bill: 30 minutes of speech is 5.1 k Gemma 4 tokens (from one Mandarin sentence’s rate; 6.1 k from one English sentence) against Mimi’s 180 k and EnCodec 6 kbps’s 1.08 M — ×35 and ×211	torch 2.14.0 + transformers 5.17.0 (no new packages); Demo 1’s three models; the W5 Demo 1 subset (dev-clean, downloaded by that demo); a text tokenizer in the Hugging Face cache for the bill	codecs on mps or CPU (the subset takes minutes, once); statistics on CPU
Demo 3 sixteen codes, four clusters, five ways to train	p14–p15	Why: collapse has two causes — a codebook initialised outside the data, and an update that only moves the chosen code — and EMA is often taken for the cure; in two dimensions all of it is visible. What you see: four Gaussian clusters inside [3, 6]^2 and sixteen codes started at \mathcal{N}(\mathbf{0}, \mathbf{I}); five ways to train them for 200 steps, each animated: gradient on the codebook loss, EMA, k-means initialisation, dead-code reset, FSQ with \Lambda = (4, 4); then the same sixteen codes started inside the data. Result (rehearsed 2026-10-18, identical to the slide’s panels): gradient 2 / 16 codes used (MSE 0.997), EMA still 2 / 16 (0.993) — EMA fixes noisy updates, not collapse; k-means init 16 / 16 (0.065); reset 16 / 16 (0.069), all fourteen dead codes revived at step 5 at once; FSQ 16 / 16 but MSE 0.123, twice k-means — equally spaced levels waste capacity on four clusters; started inside the data, both update rules keep all 16 alive, so the collapse is not tuned in	numpy, matplotlib	any laptop; everything computes in under a second

Tested on one machine — an Apple M5 Max (64 GB, no CUDA), macOS 27.0, Python 3.12 — with the versions pinned in demos/versions.lock; the same code also passes its self-test on a 2-core Linux container. Menus and labels are in Chinese; the keys are listed in the README.

References

Foundations (VQ and quantisation) - van den Oord, Vinyals & Kavukcuoglu, Neural Discrete Representation Learning (VQ-VAE), NeurIPS 2017 (arXiv:1711.00937) — the three-term loss, straight-through (“copy gradients”), and the EMA update in the appendix; this week’s first two derivations follow it - Razavi, van den Oord & Vinyals, Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019 (arXiv:1906.00446) — the precedent for hierarchical quantisation and EMA codebooks - Bengio, Léonard & Courville, Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, arXiv:1308.3432 (2013) — where the straight-through estimator is named (also cited in W5) - Mentzer, Minnen, Agustsson & Tschannen, Finite Scalar Quantization: VQ-VAE Made Simple, ICLR 2024 (arXiv:2309.15505) - Yu et al., Vector-quantized Image Modeling with Improved VQGAN, ICLR 2022 (arXiv:2110.04627) — low-dimensional factorised codes and ℓ2-normalised lookups improve codebook usage; the source of DAC’s codebook design - Dhariwal et al., Jukebox: A Generative Model for Music, arXiv:2005.00341 (2020) — random restarts for unused codes (codebook reset), adopted by SoundStream - Łańcucki et al., Robust Training of Vector Quantized Bottleneck Models, IJCNN 2020, pp. 1–7 (DOI 10.1109/IJCNN48605.2020.9207145; arXiv:2005.08520) — data-dependent (k-means) codebook initialisation and periodic re-estimation early in training - Huh, Cheung, Agrawal & Isola, Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks, ICML 2023, PMLR 202:14096–14113 (arXiv:2305.08842) — “internal codebook covariate shift”, affine reparameterisation and alternating optimisation - Juang & Gray, Multiple Stage Vector Quantization for Speech Coding, ICASSP 1982, vol. 7, pp. 597–600 (DOI 10.1109/ICASSP.1982.1171604) — the origin of multistage (residual) VQ: RVQ was not invented in 2021 - Gray & Neuhoff, Quantization, IEEE Transactions on Information Theory 44(6):2325–2383 (1998; DOI 10.1109/18.720541) — the survey of quantisation theory behind this week’s rate–distortion remark - Cover & Thomas, Elements of Information Theory, 2nd ed., Wiley 2006, Ch. 10 Rate Distortion Theory, Theorem 10.3.2 — R(\mathcal{D}) = \tfrac12\log_2(\sigma^2/\mathcal{D}) for a Gaussian source - BB (Bishop & Bishop, 2024), the chapters on VAEs and discrete latent variables

Discriminators from adversarial vocoders (where the codec training recipe comes from) - Kumar et al., MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis, NeurIPS 2019 (arXiv:1910.06711) — multi-scale waveform discriminator, feature-matching loss - Kong, Kim & Bae, HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis, NeurIPS 2020 (arXiv:2010.05646) — the multi-period discriminator; W8 returns to it

Speech codecs - Zeghidour, Luebs, Omran, Skoglund & Tagliasacchi, SoundStream: An End-to-End Neural Audio Codec, IEEE/ACM TASLP 30:495–507 (2022; arXiv:2107.03312) — residual VQ, quantizer dropout, k-means initialisation and dead-code reset, the adversarial + feature-matching + multi-scale spectral recipe; the source of “3 kbps against Opus at 12 kbps” - Défossez, Copet, Synnaeve & Adi, High Fidelity Neural Audio Compression (EnCodec), TMLR 2023 (arXiv:2210.13438) — 75 Hz, codebooks of 1024, 1.5–24 kbps, multi-scale STFT discriminator, loss balancer; the acoustic codec in Demo 1 - Kumar, Seetharaman, Luebs, Kumar & Kumar, High-Fidelity Audio Compression with Improved RVQGAN (DAC), NeurIPS 2023 (arXiv:2306.06546) — 8-dimensional ℓ2-normalised codebooks for codebook usage, multi-scale mel loss, multi-period and multi-band discriminators, the quantizer-dropout trade-off - Borsos et al. (11 authors), AudioLM: A Language Modeling Approach to Audio Generation, IEEE/ACM TASLP 31:2523–2533 (2023; DOI 10.1109/TASLP.2023.3288409; arXiv:2209.03143) — semantic tokens (k-means on w2v-BERT) stacked on acoustic tokens (SoundStream) in three stages; the experiments showing semantic tokens alone are not enough - Zhang, Zhang, Li, Zhou & Qiu, SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models, ICLR 2024 (arXiv:2308.16692) — HuBERT (layer 9 or the average of all layers) distilled into the first RVQ layer, with a continuous and a pseudo-label variant - Ye et al. (12 authors), Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (X-Codec), AAAI 2025, 39(24):25697–25705 (DOI 10.1609/aaai.v39i24.34761; arXiv:2408.17175) — SSL semantic features injected before the RVQ, with a semantic reconstruction loss; the representative paper treating “acoustic codecs lack semantics” as a defect to fix - Ji et al. (17 authors), WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling, ICLR 2025 (arXiv:2408.16532) — a single codebook (V = 4096), 40 or 75 tokens per second of 24 kHz audio (0.9 kbps at 75 Hz), k-means initialisation, an iSTFT decoder with attention - Xin, Tan, Takamichi & Saruwatari, BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec, arXiv:2409.05377 (2024) — a single codebook of 8192 codes at 1.04 kbps; another case of a low-dimensional VQ improving codebook usage - Parker et al., Scaling Transformers for Low-Bitrate High-Quality Speech Coding (TAAE), ICLR 2025 (arXiv:2411.19842) — FSQ in a speech codec, 400 / 700 bps - Siuzdak, Vocos: Closing the Gap Between Time-Domain and Fourier-Based Neural Vocoders for High-Quality Audio Synthesis, ICLR 2024 (arXiv:2306.00814) — a decoder that predicts STFT coefficients and inverts them; the decoder behind WavTokenizer - Défossez et al. (Kyutai, 8 authors), Moshi: A Speech-Text Foundation Model for Real-Time Dialogue, arXiv:2410.00037 (2024) — Mimi (split RVQ, WavLM distillation, 12.5 Hz, 1.1 kbps, 80 ms) is the most important example this week; the RQ-Transformer and the acoustic delay are left to W13. A technical report, cited as a preprint - Kyutai, the kyutai/mimi model card (Hugging Face; not a paper) — weight licence CC-BY 4.0, 12.5 Hz, 1.1 kbps (the card states neither the sampling rate nor the number of codebooks); the weights Demos 1 and 2 load - Lee, Kim, Kim, Cho & Han, Autoregressive Image Generation using Residual Quantization, CVPR 2022, pp. 11513–11522 (DOI 10.1109/CVPR52688.2022.01123; arXiv:2203.01941) — the RQ-Transformer: autoregression over the N_q codes within one time step; the architecture Moshi builds on (W13) - Copet et al. (8 authors), Simple and Controllable Music Generation (MusicGen), NeurIPS 2023 (Advances in NeurIPS 36, pp. 47704–47720; arXiv:2306.05284) — the delay pattern over codebooks; another answer to “one time step carries N_q tokens” (W13)

Evaluation and analysis - Wu et al., Codec-SUPERB: An In-Depth Analysis of Sound Codec Models, Findings of ACL 2024, pp. 10330–10348 (DOI 10.18653/v1/2024.findings-acl.616) — a benchmark that measures reconstruction and downstream tasks separately - Shi et al., ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech, SLT 2024, pp. 562–569 (DOI 10.1109/SLT61566.2024.10832289) - Mousavi et al. (8 authors), DASB — Discrete Audio and Speech Benchmark, TMLR 2026 (arXiv:2406.14294, first version 2024) — semantic versus acoustic (compression) tokens across discriminative and generative tasks; from the abstract, “semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features” - Park, Takamichi, Chan, Kando, Saito & Saruwatari, Analysing the Language of Neural Audio Codecs, ASRU 2025 (arXiv:2509.01390) — Zipf, Heaps and entropy / redundancy statistics of codec tokens and their correlation with WER / UTMOS; its extension Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions (arXiv:2608.31037, submitted to TASLP) adds codebook occupancy and JSD across 13 codecs, three corpora and three noise conditions; a graduate-level discussion topic - Saloev, Ballier & Pacquetet, Probing the phon* Representations of the Mimi Codebooks, ICNLSP 2026, pp. 322–331 — what segmental information Mimi’s first codebook encodes (English, TIMIT); the method transfers directly to tone - Osakuade & King, Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yorùbá, Speech Prosody 2026, pp. 38–42 (DOI 10.21437/SpeechProsody.2026-8) — continuous SSL representations encode tone, discrete units lose it faster than segmental information; the first peer-reviewed evidence on this course’s tone thread

2025–2026 frontier (further reading or class discussion) - Chen et al. (12 authors), SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization, arXiv:2510.16841 (2025, preprint) — a frozen 12.5 Hz semantic stream plus a 25 / 50 Hz acoustic stream, 525 / 875 bps - Li et al. (10 authors), FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates, ICLR 2026 (arXiv:2510.00981) — merges semantically similar frames; 3–12.5 Hz controllable at inference - Wang, Guo, Shao, Li & Yu, CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate, ICASSP 2026 (arXiv:2506.21074) — an architecture-agnostic dynamic-frame-rate plug-in; an 80 Hz backbone run at 40 Hz - Della Libera, Subakan & Ravanelli, Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization (DyCAST), arXiv:2601.23174 (2026, preprint) — tokens roughly aligned to characters at 6–18 Hz; a direct answer to the “letters do not align” failure of the alphabet metaphor - Gichamba & Busogi, Probing Low Frame Rate Degradation in Neural Audio Codecs, Interspeech 2026 (arXiv:2606.16969) — the quality cliff at 6.25 Hz is traced to the training configuration (too few tokens per fixed-length clip) rather than to the frame rate itself

Standards and textbooks - Valin, Vos & Terriberry, RFC 6716: Definition of the Opus Audio Codec, IETF Proposed Standard, 2012 (updated by RFC 8251) — a standard, not a paper; frames of 2.5–60 ms and algorithmic delays of 5–65.2 ms (§2); the reference point for “neural codecs versus Opus” - JM3 Ch 16 Automatic Speech Recognition (2026-08-19 release) — background on the ASR side; the main readings this week are SoundStream and §3 of the Moshi report - Whether acoustic codecs preserve lexical tone (further searching; no peer-reviewed study found as of 2026-10-16): neural audio codec lexical tone Mandarin, Mimi codebook tone probing, low bitrate codec tone preservation - New directions in 2026 (further searching): dynamic frame rate speech tokenizer, syllable-aligned speech tokens, single-codebook speech codec LLM