W2｜Speech Signals, Auditory Front-Ends, and the Physics of Representation

Part I — Foundations · 語音訊號、聽覺前端與表徵的物理基礎

▶ Slides for this week

Where This Fits

This week supplies the input-side language that every later module speaks. For students with no prior background it is the only systematic pass through DSP in the whole semester, and the only systematic pass through phonetics, so it is kept deliberately narrow: only what will be used later. On the signal side that means sampling, the STFT, mel, phase, F0 and SNR — not filter design in general. On the phonetics side it means phones versus phonemes, how consonants are articulated, coarticulation, and the three acoustic correlates of prosody (F0, duration, energy).

That last item matters more in this course than in most. Mandarin, Taiwanese Hokkien and Hakka are tone languages: F0 carries lexical content and intonation on the same curve. The consequences come back repeatedly — when we tokenise speech (W6), when we try to control synthesis (W9), and when we look for turn-taking cues (W12).

Learning Objectives

  1. Derive the time–frequency resolution trade-off of the STFT, and keep apart three quantities that are routinely conflated: window length sets frequency resolution and lookahead, hop sets the frame rate, and the FFT size only sets the bin spacing.
  2. Explain how the source–filter model separates the excitation (F0, voicing) from the vocal-tract filter (formants, spectral envelope) in the frequency domain, and say where that separation breaks — including why “source = who is speaking, filter = what is said” does not hold in a tone language.
  3. Motivate each step of log-mel and MFCC (why the log, why mel, why the DCT), explain why MFCC has largely given way to log-mel in the deep-learning era, and point out where two implementations of “log-mel” can differ.
  4. Split the distortion of “waveform from spectrogram” into its three sources — phase, mel compression, and over-smoothed predicted spectra — and use STFT consistency to explain why Griffin-Lim is possible and why it is not enough.
  5. List the three acoustic correlates of prosody, explain how lexical tone and intonation share F0, and predict what that implies for methods developed mainly on English: semantic tokens, prosody control, and prosodic turn-taking cues.

Demos

Two of this week’s three demos are programs; they are published so that you can read them, re-run them, and change them — you are not required to run anything. Each folder’s README.md gives requirements, setup, commands, and what you should see; the index at demos/ explains the shared environment.

Slides What it shows Needs Runs on
Demo 1
live spectrogram, pitch track, vowel chart		 p12–p13, p24, p54 Why: the source–filter model, live: the vowel lives in the formants, the pitch in F_0 — move one and watch which numbers follow.
What you see: a microphone, a 4 s scrolling spectrogram switchable between a 5 ms and a 30 ms window, an autocorrelation F_0 track and an LPC (F_1, F_2) dot on a vowel chart; used for vowels and the glide between them, for one sound under two window lengths, and for Mandarin tones and then a question. A Praat script for exact readings is included.
Result: the dot follows the mouth, and sits only roughly on the reference corners (1952 averages of American English adult men); the 5 ms window shows glottal pulses as vertical stripes, the 30 ms window harmonics as horizontal lines; a pitch ceiling set too low gives an octave error		 numpy + scipy + matplotlib + sounddevice, a microphone; all DSP in demos/dsp.py any laptop
Demo 2
phase × magnitude: the 2 × 3 reconstruction grid		 p35–p36, p42, p44 Why: the spectrogram drops the phase and the mel filterbank drops more; this pulls the two losses apart so each can be heard alone.
What you hear: one 5 s sentence re-synthesised from linear or mel-80 magnitude crossed with the original, a Griffin–Lim or a random phase, plus mel-40/20/10 with the original phase; Griffin–Lim’s iteration count as a knob. Least-squares ISTFT, the 1984 Griffin–Lim without momentum, no librosa.
Result: Griffin–Lim brings the log-spectral distance from 8.47 dB (random phase) to 1.71 dB at 256 iterations while the waveform SNR stays between −1 and −4 dB; with mel-80 in the chain the distance floors near 6 dB — two separate costs		 numpy + scipy + matplotlib; the lecturer’s recording and its reconstructions are not published — record your own 5 s sentence any laptop
Demo 3
front-end mismatch		 p60 Why: two front-ends that are both called “80-dim log-mel, 25 ms / 10 ms” can still differ; does a pretrained recogniser notice?
What you would see: the same audio and the same pretrained recogniser, only the mel recipe changed (HTK vs. Slaney, filterbank normalisation on or off), WER compared.
Result: not run yet — the slide presents it as a prediction question		 — —

Tested on one machine only — an Apple M5 Max (64 GB, no CUDA), macOS 26.6, Python 3.12 — with the versions pinned in demos/versions.lock. Menus and labels are in Chinese; the keys are listed in each README.

References

  • JM3 Ch 15 Phonetics and Speech Feature Extraction — the reference for both halves of the week, phonetics and feature extraction
  • HAH Ch 2 (Spoken Language Structure), Ch 5 (Digital Signal Processing), Ch 6 (Speech Signal Representations) — the full derivations of the cepstrum, LPC and the source–filter model are here
  • Griffin & Lim, Signal Estimation from Modified Short-Time Fourier Transform, IEEE TASSP, 1984
  • Nawab, Quatieri & Lim, Signal Reconstruction from Short-Time Fourier Transform Magnitude, IEEE TASSP 31(4):986–998, 1983 — the conditions under which the STFT magnitude alone determines the signal
  • Davis & Mermelstein, Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences, IEEE TASSP 28(4):357–366, 1980 — the original MFCC paper
  • Stevens, Volkmann & Newman, A Scale for the Measurement of the Psychological Magnitude Pitch, JASA 8(3):185–190, 1937 — where the mel scale comes from. Read it to see that mel is the outcome of a small perceptual experiment, not a physical quantity
  • de Cheveigné & Kawahara, YIN, a Fundamental Frequency Estimator for Speech and Music, JASA 111(4):1917–1930, 2002
  • Mauch & Dixon, pYIN: A Fundamental Frequency Estimator Using Probabilistic Threshold Distributions, ICASSP 2014
  • Kim, Salamon, Li & Bello, CREPE: A Convolutional Representation for Pitch Estimation, ICASSP 2018
  • F0 modelling in tone languages (further searching): tone sandhi Taiwanese Southern Min acoustic, lexical tone discrete speech units, tonal language self-supervised speech representation tone, tone intonation interaction F0
  • SI-SDR, definition and motivation: SDR half-baked or well done (used formally in W10)
  • LPC meets neural networks: LPCNet linear prediction neural vocoder
  • Front-ends and deep learning: learnable filterbank speech, SincNet, LEAF frontend learnable audio
回到頂端