AI Engineering/Phase 06/Lesson 12

Build a Voice Assistant Pipeline — The Phase 6 Capstone

Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research problem — but the integration details decide whether it ships.

Build 2 hr Python

Prerequisites: Phase 6 · 04, 05, 06, 07, 11; Phase 11 · 09 (Function Calling); Phase 14 · 01 (Agent Loop)

#The Problem

Build an end-to-end assistant:

Captures mic input (16 kHz mono).
Detects start/end of user speech.
Transcribes streaming.
Passes transcript to an LLM that can call tools (timer, weather, calendar).
Streams LLM text to a TTS.
Plays audio back to the user.
Stops if the user interrupts mid-response.

Latency target: first TTS audio byte within 800 ms of the user finishing their utterance on a laptop CPU. Quality target: no missed words, no hallucinated subtitles on silence, no voice cloning leakage, no prompt injection success.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.