W5｜Self-Supervised Representation Learning: Where Speech Foundation Models Come From

Part I — Foundations · 自監督表徵學習（SSL）：語音基礎模型的來源

▶ Slides for this week

Where This Fits

W4 opened the encoder — the shape of a Transformer / Conformer and the price of streaming — but never said where its ability comes from. Every loss in W3 and W4 assumed a transcript sitting next to the audio. This week removes the transcript and asks what an encoder can learn without labels, and how. It is the hub of the course’s representation axis: W6’s semantic tokens are direct descendants of this week’s k-means units (Mimi distils WavLM into its first codebook), W7’s ASR encoders are mostly this week’s models with a CTC layer fine-tuned on top, and the first audio-native language model in W11 (GSLM) was trained on exactly these discrete units.

The main thread is how the target is built. A masked language model over text has natural discrete words to guess; speech has none — so the differences between CPC, wav2vec 2.0, HuBERT, data2vec and BEST-RQ lie almost entirely in “what is the answer for the masked frame”: a latent the model quantises itself and contrasts against negatives, an offline k-means cluster id, the continuous representation of an EMA teacher, or a completely random projection and codebook. Who builds the target, and whether it moves during training, decides whether the model can collapse, whether negatives are needed, and whether the procedure can be iterated. The second thread is layers: the twelve layers of one model are not a ladder that improves monotonically — speaker information sits low, phonetic information in the middle, and in a pre-trained (not fine-tuned) wav2vec 2.0 the top layers turn back towards the input. Which layer a downstream task reads is a representation decision, not an implementation detail. The third thread is how to read a comparison: comparisons between SSL models are almost always confounded by the amount and domain of the pre-training data; this is the most common reading error in the field.

Where the three course-long axes sit this week. Representation: the middle of W2’s trio (log-mel / SSL feature / discrete token) is filled in. The frame rate is fixed by the stride of the convolutional front end (320 samples at 16 kHz, 20 ms — W2’s \Delta times s = 2), and it is fixed before any downstream task exists; this week’s k-means units are the first form of the third representation. Latency: a standard SSL encoder is a bidirectional Transformer pre-trained on whole utterances; putting it into a streaming system costs what W4 described (causal or chunked variants — the BEST-RQ paper evaluates a streaming setting itself), and a few hundred million parameters per frame is a new term in D_{\text{comp}}. Supervision: this is the first week in which the answer to “where does the supervision come from” changes. Pre-training needs no labels, but fine-tuning and evaluation still do; SSL changes the marginal value of a labelled hour (the wav2vec 2.0 abstract sells ten minutes of labels), it does not remove labels. And “the model builds its own target” continues into W6’s codecs and W11’s unit language models.

The local-language thread takes a concrete shape this week. The mainstream SSL models are pre-trained on English (LibriSpeech / Libri-light), and their discrete units are designed to carry “content” while F0 is extracted as a separate stream (the resynthesis architecture of Polyak et al. 2021 is exactly units + prosody + speaker). For English that is a sensible disentanglement; for Mandarin, Taiwanese or Hakka it removes part of the lexical content — tone — from the units. This is the SSL-level version of W2’s forecast that “semantic tokens flatten intonation”, and it remains a prediction to be verified (multilingual models and ML-SUPERB’s 143 languages exist, but whether tone survives in the units needs a dedicated analysis; this week leaves a search pointer).

Learning Objectives

  1. Derive InfoNCE as a lower bound on mutual information (substitute the optimal critic f^* \propto p(\mathbf{z}\mid\mathbf{c})/p(\mathbf{z}) back into the loss to obtain I(\mathbf{z};\mathbf{c}) \geq \ln N - \mathcal{L}_N), point out that the bound can never exceed \ln N, and explain why CPC predicts the future while wav2vec 2.0 predicts the masked quantised latent instead.
  2. Compare four ways of building the target — wav2vec 2.0 (contrastive, with an online Gumbel-VQ target), HuBERT (offline k-means cluster ids, refined by iteration), data2vec (multi-layer continuous targets from an EMA teacher), BEST-RQ (a frozen random-projection quantiser) — and state for each who builds the target, whether it moves, what prevents collapse, and how it fails.
  3. Write down the Gumbel-softmax reparameterisation with its straight-through gradient, show that wav2vec 2.0’s diversity loss maximises the entropy of codebook usage, and explain why codebook collapse returns in a harsher form in W6.
  4. Describe the layer structure of SSL representations (speaker information low, phonetic information in the middle, and top layers of a pre-trained model that may turn back towards the input), measure it with CCA or linear probes, use it to choose a layer for a downstream task, and explain what SUPERB’s learned layer weights \mathbf{w} do and do not tell you.
  5. Read an SSL comparison critically: show how pre-training data size, data domain and model size confound architectural claims, say which kinds of models a frozen-upstream benchmark favours, and explain why HuBERT’s discrete units are not phonemes (finer, context-dependent, redundant; speaker information reduced but not gone; F0 largely absent).

Demos

All three demos are programs and are published so that you can read them, re-run them, and change them — you are not required to run anything. Each folder’s README.md gives requirements, setup, commands, and what you should see; the index at demos/ explains the shared environment.

Slides What it shows Needs Runs on
Demo 1
layer-wise probing: three curves across thirteen layers		 p32–p33 Why: an SSL encoder gives thirteen representations; which layer holds the speaker, which the phones, and which still resembles the input decides which layer a task should read.
What you see: HuBERT-base and wav2vec 2.0 base (pre-trained, not fine-tuned) on all 40 dev-clean speakers × 10 utterances; for each of the 13 layers a frame-level speaker probe, a phone probe (MFA labels) and PWCCA with layer 0, drawn point by point.
Result: the speaker probe peaks at layer 2 (0.94 / 0.89), falls to layer 8 and comes back at the top (0.78 / 0.94 — wav2vec 2.0’s top layer is its best speaker layer); phones peak at layer 11 for HuBERT (0.81) and layer 9 for wav2vec 2.0 (0.80, top 0.56); PWCCA with layer 0 dips in the middle and turns back at the top, more for wav2vec 2.0 — the last layer is not the best for everything		 torch 2.14.0 + transformers 5.17.0 + pyarrow (fetch only); two ≈ 380 MB models; dev-clean.tar.gz encoders on mps or CPU, probes on CPU; feature extraction cached (≈ 350 MB per model)
Demo 2
units ≠ phones: three teachers’ purities, and one sentence aligned		 p40–p42 Why: HuBERT’s k-means targets are often called phone-like units; this measures how phone-like, for three teachers and for one sentence.
What you see: k-means (K = 100, numpy) on the 23,726 phone-labelled frames of Demo 1 for MFCC, HuBERT-base layer 6 and layer 9, scored with the HuBERT paper’s three measures on held-out frames; then W3’s sentence with its words and MFA phones over its unit sequence, and mister said by four speakers.
Result: PNMI 0.30 for MFCC, 0.60 for layer 6, 0.58 for layer 9 — the model’s own layers cluster into phones twice as well as the MFCC that taught it, and layer 9 is not better than layer 6 here; on layer 6 a phone segment takes 2.6 units, unit edges fall on phone edges 25 % of the time against 22 % by chance, and the four misters share about half their runs		 shares Demo 1’s caches and torch + transformers; pyarrow for fetch only; no new packages encoder on mps or CPU for four utterances; k-means on CPU (≈ 25 s)
Demo 3
the ceiling at \ln N		 p12–p14 Why: InfoNCE’s estimate is capped at \ln N; data with a known mutual information shows what that ceiling does to a trained critic.
What you see: two 8-dimensional Gaussian vectors with correlation 0.7 per dimension (mutual information 2.693 nat); a critic trained at each N = 2, 4, \dots, 1024, its estimate \ln N - \mathcal{L}_N plotted against \ln N and the truth.
Result: at N = 8 the estimate is 1.54 under a ceiling of 2.08 — it cannot reach 2.69 however long it trains; every estimate stays below \ln N and below the truth; from 64 to 1024 negatives the estimate gains only 0.31 nat while the spread over batches falls from 0.20 to 0.06		 numpy, matplotlib any laptop; the full sweep takes 30 s on the M5 Max, ≈ 70 s on a 2-core container

Tested on one machine — an Apple M5 Max (64 GB, no CUDA), macOS 27.0, Python 3.12 — with the versions pinned in demos/versions.lock; the same code also passes its self-test on a 2-core Linux container. Menus and labels are in Chinese; the keys are listed in the README.

References

  • Oord, Li & Vinyals, Representation Learning with Contrastive Predictive Coding, arXiv:1807.03748 (2018) — CPC and InfoNCE; this week’s derivation follows it, with the rigorous version in the next entry
  • Poole, Ozair, van den Oord, Alemi & Tucker, On Variational Bounds of Mutual Information, ICML 2019, PMLR 97:5171–5180 (arXiv:1905.06922) — a proof of the InfoNCE bound that needs neither of CPC’s approximations, its ceiling at the log of the batch size, and its bias–variance trade-off against other variational bounds (InfoNCE: low variance, high bias; NWJ: unbiased, high variance)
  • Schneider, Baevski, Collobert & Auli, wav2vec: Unsupervised Pre-Training for Speech Recognition, Interspeech 2019, pp. 3465–3469 (DOI 10.21437/Interspeech.2019-1873; arXiv:1904.05862) — purely convolutional contrastive pre-training, the predecessor of wav2vec 2.0
  • Baevski, Schneider & Auli, vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations, ICLR 2020 (arXiv:1910.05453) — the first use of Gumbel-softmax / k-means quantisation in speech SSL, followed by BERT on the discrete codes
  • Baevski, Zhou, Mohamed & Auli, wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, NeurIPS 2020 (Advances in NeurIPS 33; arXiv:2006.11477) — contrastive target, Gumbel-VQ, diversity loss; the two same-architecture, different-data rows come from this paper’s results table
  • Jang, Gu & Poole, Categorical Reparameterization with Gumbel-Softmax, ICLR 2017 (arXiv:1611.01144) — the reparameterisation in this week’s maths
  • Bengio, Léonard & Courville, Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, arXiv:1308.3432 (2013) — where the straight-through estimator is named (stated in the abstract)
  • Hsu, Tsai, Bolte, Salakhutdinov & Mohamed, HuBERT: How Much Can a Bad Teacher Benefit ASR Pre-Training?, ICASSP 2021 — the experiments behind “consistency matters more than correctness”; the conference version of the line continued in the next entry
  • Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov & Mohamed, HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, IEEE/ACM TASLP 29:3451–3460 (2021; DOI 10.1109/TASLP.2021.3122291; arXiv:2106.07447) — offline k-means targets, loss over masked frames only, iteration; the definitions of phone purity, cluster purity and PNMI
  • Hsu, Sriram, Baevski, Likhomanenko, Xu, Pratap, Kahn, Lee, Collobert, Synnaeve & Auli, Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training, Interspeech 2021, pp. 721–725 (DOI 10.21437/Interspeech.2021-236; arXiv:2104.01027) — systematic experiments with pre-training, fine-tuning and test data from different domains; the source of “unlabelled target-domain data in pre-training”
  • Chen et al. (19 authors), WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing, IEEE JSTSP 16(6):1505–1518 (2022; DOI 10.1109/JSTSP.2022.3188113; arXiv:2110.13900) — denoising and overlapped-speech auxiliary tasks, gated relative position bias, 60k → 94k hours
  • Baevski, Hsu, Xu, Babu, Gu & Auli, data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language, ICML 2022, PMLR 162:1298–1312 (arXiv:2202.03555) — EMA teacher, multi-layer continuous targets
  • Grill et al. (14 authors), Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning, NeurIPS 2020 (arXiv:2006.07733) — the origin of the EMA-teacher design without negatives; the empirical answer to collapse
  • Chiu, Qin, Zhang, Yu & Wu, Self-supervised Learning with Random-projection Quantizer for Speech Recognition (BEST-RQ), ICML 2022, PMLR 162:3915–3924 (arXiv:2202.01855) — a frozen random quantiser; the abstract states the streaming and multilingual settings
  • Zhang et al. (27 authors), Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages, arXiv:2303.01037 (2023) — BEST-RQ-style random-projection quantisation scaled to 12 million hours and 300+ languages; a technical report, cited as a preprint
  • Yang et al. (20 authors), SUPERB: Speech Processing Universal PERformance Benchmark, Interspeech 2021, pp. 1194–1198 (DOI 10.21437/Interspeech.2021-1775; arXiv:2105.01051) — frozen upstream, light downstream heads, layer-wise weighted sum
  • Shi et al. (11 authors), ML-SUPERB: Multilingual Speech Universal PERformance Benchmark, Interspeech 2023, pp. 884–888 (DOI 10.21437/Interspeech.2023-1316; arXiv:2305.10615) — 143 languages; the abstract states that multilingual models do not always beat monolingual ones
  • Huang et al., Dynamic-SUPERB: Towards a Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech, ICASSP 2024, pp. 12136–12140 (DOI 10.1109/icassp48485.2024.10448257)
  • Mohamed et al. (12 authors), Self-Supervised Speech Representation Learning: A Review, IEEE JSTSP 16(6):1179–1210 (2022; DOI 10.1109/JSTSP.2022.3207050; arXiv:2205.10643) — the generative / contrastive / predictive taxonomy; this week’s survey reading
  • Pasad, Chou & Livescu, Layer-wise Analysis of a Self-supervised Speech Representation Model, ASRU 2021 (arXiv:2107.04734) — CCA, mutual information and probes across the layers of wav2vec 2.0; the source for top layers turning back towards the input
  • Pasad, Shi & Livescu, Comparative Layer-wise Analysis of Self-Supervised Speech Models, ICASSP 2023 (arXiv:2211.03929) — layer-wise comparison across models; “a single well-chosen layer often matches a weighted sum of all layers”
  • Lakhotia et al. (11 authors), On Generative Spoken Language Modeling from Raw Audio, TACL 9:1336–1354 (2021; DOI 10.1162/tacl_a_00430; arXiv:2102.01192) — GSLM: language models trained on the discrete units (50 / 100 / 200) of CPC, wav2vec 2.0 and HuBERT; the starting point of W11
  • Polyak, Adi, Copet, Kharitonov, Lakhotia, Hsu, Mohamed & Dupoux, Speech Resynthesis from Discrete Disentangled Self-Supervised Representations, Interspeech 2021, pp. 3615–3619 (DOI 10.21437/Interspeech.2021-475; arXiv:2104.00355) — units + prosody + speaker as three streams; the basis for “F0 is largely absent from the units, speaker information reduced but not gone”
  • Sicherman & Adi, Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling, ICASSP 2023 (arXiv:2301.00591) — units correlate strongly with phonemes but are redundant, and correlate weakly with speaker; clustering improved with a redundancy measure
  • JM3 Ch 16 Automatic Speech Recognition (2026-08-19 release) — background on the ASR side; the main reading this week is the review by Mohamed et al.
  • Whether tone survives in SSL units and representations (further searching): lexical tone self-supervised speech representation probing, HuBERT units Mandarin tone, tonal language discrete speech units
  • New directions in 2026 (further searching): JEPA speech tokenizer, speech SSL scaling laws, SSL robustness spontaneous conversational speech
回到頂端