Audio-Language Models: the Whisper to Audio Flamingo 3 Arc

Whisper (Radford et al., December 2022) settled speech recognition — 680k hours of weakly-supervised multilingual speech, a simple encoder-decoder transformer, a benchmark that made every subsequent ASR release cite it. But recognition is not reasoning. Asking "what instruments are in this recording" or "what emotion is the speaker expressing" or "what happened at minute 3" requires audio understanding, not transcription. Qwen-Audio, SALMONN, LTU, and NVIDIA's Audio Flamingo 3 (AF3, July 2025) progressively built that stack: keep Whisper-class encoders, bolt on Q-formers, train on audio-text instruction data, add chain-of-thought reasoning. This lesson walks the arc.

Build 3 hr Python (stdlib, log-Mel spectrogram + audio Q-former skeleton)

Prerequisites: Phase 6 (Speech and Audio), Phase 12 · 03 (Q-Former)

#Learning Objectives

Compute a log-Mel spectrogram from a waveform: windowing, FFT, filter banks, log transform.
Compare encoder options: Whisper encoder, BEATs, AF-Whisper hybrid. When each wins.
Build an audio Q-former: N learnable queries cross-attending to spectrogram patches.
Explain cascaded (Whisper-then-LLM) vs end-to-end audio-LLM training: why end-to-end scales better for reasoning.

#Learning Objectives

Keep going. The rest of this lesson is one step away.