Sign inStart free
AI Engineering/Phase 06
Core

Speech and Audio

Signal processing, speech recognition and synthesis from first principles.

17 lessons ~18.5 hr
  1. Audio Fundamentals — Waveforms, Sampling, Fourier TransformLearn · Free · 45 minWaveforms are the raw signal. Spectrograms are the representation. Mel features are the ML-friendly form. Every modern ASR and TTS pipeline walks this ladder, and the first rung is understanding sampling and Fourier.Open
  2. Spectrograms, Mel Scale & Audio FeaturesBuild · 45 minNeural nets do not consume raw waveforms well. They consume spectrograms. They consume mel spectrograms even better. Every ASR, TTS, and audio classifier in 2026 lives or dies by this single preprocessing choice.Needs Full Access
  3. Audio Classification — From k-NN on MFCCs to AST and BEATsBuild · 1 h 15 minEverything from "dog barking vs siren" to "which language is this" is audio classification. The features are mels. The architecture moves each decade. The evaluation stays AUC, F1, and per-class recall.Needs Full Access
  4. Speech Recognition (ASR) — CTC, RNN-T, AttentionBuild · 45 minSpeech recognition is audio classification at every timestep, glued together by a sequence model that knows English and silence. CTC, RNN-T, and attention are the three ways to do it. Pick one and understand why.Needs Full Access
  5. Whisper — Architecture & Fine-TuningBuild · 1 h 15 minWhisper is a 30-second-window transformer encoder-decoder, trained on 680k hours of multilingual weakly-supervised audio-text pairs. One architecture, multiple tasks, robust across 99 languages. The 2026 reference ASR.Needs Full Access
  6. Speaker Recognition & VerificationBuild · 45 minASR asks "what did they say?" Speaker recognition asks "who said it?" The math looks the same — embeddings plus cosine — but every production decision hinges on a single EER number.Needs Full Access
  7. Text-to-Speech (TTS) — From Tacotron to F5 and KokoroBuild · 1 h 15 minASR inverts speech to text; TTS inverts text to speech. The 2026 stack is three parts: text → tokens, tokens → mel, mel → waveform. Each part has a default model that fits in a laptop.Needs Full Access
  8. Voice Cloning & Voice ConversionBuild · 1 h 15 minVoice cloning reads your text in someone else's voice. Voice conversion rewrites your voice into someone else's while preserving what you said. Both hang on the same decomposition: separate speaker identity from content.Needs Full Access
  9. Music Generation — MusicGen, Stable Audio, Suno, and the Licensing EarthquakeBuild · 1 h 15 min2026 music generation: Suno v5 and Udio v4 dominate commercial; MusicGen, Stable Audio Open, and ACE-Step lead open-source. The technical problem is mostly solved. The legal problem (Warner Music $500M settlement, UMG settlement) reshaped the field in 2025-2026.Needs Full Access
  10. Audio-Language Models — Qwen2.5-Omni, Audio Flamingo, GPT-4o AudioLearn · 45 min2026 audio-language models reason over speech + environmental sound + music. Qwen2.5-Omni-7B matches GPT-4o Audio on MMAU-Pro. Audio Flamingo Next beats Gemini 2.5 Pro on LongAudioBench. The gap between open and closed is essentially closed — except on multi-audio tasks, where everyone is near random.Needs Full Access
  11. Real-Time Audio ProcessingBuild · 1 h 15 minBatch pipelines process a file. Real-time pipelines process the next 20 milliseconds before the next 20 arrive. Every conversational AI, broadcast studio, and telephony bot lives and dies by this latency budget.Needs Full Access
  12. Build a Voice Assistant Pipeline — The Phase 6 CapstoneBuild · 2 hEverything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research problem — but the integration details decide whether it ships.Needs Full Access
  13. Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC and the Semantic-Acoustic SplitLearn · 1 h2026 audio generation is almost all tokens. EnCodec, SNAC, Mimi, and DAC turn continuous waveforms into discrete sequences that a transformer can predict. The semantic-vs-acoustic token split — first-codebook as semantic, rest as acoustic — is the most important architectural shift since the Transformer for audio.Needs Full Access
  14. Voice Activity Detection & Turn-Taking — Silero, Cobra, and the Flush TrickBuild · 45 minEvery voice agent lives or dies on two decisions: is the user speaking now, and are they done? VAD answers the first. Turn-detection (VAD + silence-hangover + semantic endpoint model) answers the second. Get either wrong and your assistant either cuts users off or never shuts up.Needs Full Access
  15. Streaming Speech-to-Speech — Moshi, Hibiki, and Full-Duplex DialogueLearn · 1 h 15 min2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks simultaneously at 200 ms latency. Hibiki does speech-to-speech translation chunk-by-chunk. Both abandon the ASR → LLM → TTS pipeline for a unified full-duplex architecture over Mimi codec tokens. This is the new reference design.Needs Full Access
  16. Voice Anti-Spoofing & Audio Watermarking — ASVspoof 5, AudioSeal, WaveVerifyBuild · 1 h 15 minVoice cloning shipped faster than defenses. 2026 production voice systems need two things: a detector (AASIST, RawNet2) that classifies real vs fake speech, and a watermark (AudioSeal) that survives compression and editing. Ship both or do not ship voice cloning.Needs Full Access
  17. Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open LeaderboardsLearn · 1 hYou cannot ship what you cannot measure. This lesson names the 2026 metrics for every audio task: ASR (WER, CER, RTFx), TTS (MOS, UTMOS, SECS, WER-on-ASR-round-trip), audio-language (MMAU, LongAudioBench), music (FAD, CLAP), and speaker (EER). Plus the leaderboards where you compare.Needs Full Access