LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model

Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like Video-LLaVA and Video-LLaMA. Each won its benchmark and failed at the others. LLaVA-OneVision argued a single curriculum could train one model to dominate all three scenarios, and that the emergent task-transfer effects (single-image skills exported to video, multi-image reasoning exported to single-image) beat the sum of specialists. The recipe is deceptively simple: a visual-token budget that stays constant across scenarios, plus an explicit curriculum that moves from single-image to OneVision (multi-image) to video. This lesson reads the budget, the curriculum, and the emergent behaviors.

Build 3 hr Python (stdlib, token budget solver + curriculum planner)

Prerequisites: Phase 12 · 05 (LLaVA), Phase 12 · 06 (any-resolution)

#Learning Objectives

Design a visual-token budget that holds constant across single-image, multi-image, and video inputs.
Order a training curriculum that transfers skills from single-image to video without catastrophic forgetting.
Explain why a single model beats specialists at the same parameter count when curriculum is done right.
Name the three emergent capabilities reported by LLaVA-OneVision: multi-camera reasoning, set-of-mark prompting, iPhone-screenshot agent.

#Learning Objectives

Keep going. The rest of this lesson is one step away.