Long-Video Understanding at Million-Token Context

A 1-hour 4K video at 24 FPS, patched and embedded, produces on the order of 60 million tokens. A 2-hour podcast episode transcribed is 30,000 tokens. A full Blu-ray feature film, even compressed with aggressive pooling, is hundreds of thousands of tokens. Google's Gemini 1.5 (March 2024) opened this era with a 10-million-token context, doing reliable needle-in-a-haystack recall over hour-long videos. LWM (Liu et al., February 2024) showed ring attention's scaling path. LongVILA and Video-XL scaled ingestion further. VideoAgent swapped raw context for agentic retrieval. Each approach is a different trade-off on compute, recall, and engineering complexity. This lesson reads them side by side.

Build 3 hr Python (stdlib, needle-in-haystack simulator + agentic-retrieval router)

Prerequisites: Phase 12 · 17 (video temporal tokens)

#Learning Objectives

Compute total visual-token counts for long-form video at varying FPS and pooling.
Explain the three scaling paths: brute context (Gemini 1.5), ring attention (LWM), token compression (LongVILA / Video-XL).
Compare raw-context video VLMs vs agentic-retrieval video VLMs (VideoAgent) on accuracy and latency.
Design a needle-in-a-haystack test for a 30-minute video and measure recall at a specific minute.

#Learning Objectives

Keep going. The rest of this lesson is one step away.