MIO and Any-to-Any Streaming Multimodal Models

GPT-4o ships a product most open models cannot replicate: an agent that hears voice, sees video, and speaks back in real time. The open-ecosystem answer by late 2024 was MIO (Wang et al., September 2024). MIO tokenizes text, image, speech, and music, trains one causal transformer over the interleaved sequences, and generates any modality to any modality. AnyGPT (Zhan et al., February 2024) was the proof of concept; MIO is the scale-up; Unified-IO 2 (Allen AI, December 2023) is the cousin with vision + action grounding. This lesson reads the any-to-any pattern — four tokenizers, one transformer, streaming-friendly decode.

Learn 2 hr Python (stdlib, four-modality token allocator + streaming decode loop)

Prerequisites: Phase 12 · 11 (Chameleon), Phase 6 (Speech and Audio)

#Learning Objectives

Design a shared vocabulary that hosts text, image, speech, and music tokens without collisions.
Compare SEED-Tokenizer (images) and SpeechTokenizer residual-VQ (speech) on compression + reconstruction trade-offs.
Explain the four-stage curriculum that builds up any-to-any generation.
Name the three open any-to-any recipes and their main trade-offs: MIO, AnyGPT, Unified-IO 2.

#Learning Objectives

Keep going. The rest of this lesson is one step away.