Video-Language Models: Temporal Tokens and Grounding

Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that an image model cannot represent. Video-LLaMA (Zhang et al., June 2023) shipped the first open video-LLM with audio-visual grounding. VideoChat and Video-LLaVA scaled the pattern. By 2025 Qwen2.5-VL's TMRoPE closed the gap with frontier proprietary models. Each system solved temporal tokens differently — Q-former per clip, concat-pool per frame, TMRoPE per token. This lesson reads the patterns, builds a uniform-vs-dynamic frame sampler, and evaluates on temporal grounding tasks.

Build 3 hr Python (stdlib, frame sampler + temporal-grounding evaluator)

Prerequisites: Phase 12 · 08 (LLaVA-OneVision)

#Learning Objectives

Explain why temporal positional encoding changes video VLM performance independently of the vision encoder.
Compare uniform, dynamic-FPS, and event-driven frame sampling on tokens-per-second vs grounding accuracy.
Describe Q-former-per-clip (Video-LLaMA) vs pooled-per-frame (Video-LLaVA) vs M-RoPE-per-token (Qwen2.5-VL) designs.
Name the four video benchmarks: VideoMME, TempCompass, EgoSchema, Video-MMMU.

#Learning Objectives

Keep going. The rest of this lesson is one step away.