Sign inStart free
AI Engineering/Phase 12/Lesson 07

Open-Weight VLM Recipes: What Actually Matters

The 2024-2026 open-weight VLM literature is a forest of ablation tables. Apple's MM1 tested 13 combinations of image encoder, connector, and data mix. Allen AI's Molmo proved detailed human captions beat GPT-4V distillation. Cambrian-1 ran 20+ encoder comparisons. Idefics2 formalized the five-axis design space. Prismatic VLMs compared 27 training recipes on a controlled benchmark. Out of all that noise, a small set of results holds across papers: image encoder matters more than connector architecture, data mixture matters more than either, and detailed human captions beat distilled synthetic data. This lesson reads those tables so you do not have to.

Learn + lab 3 hr Python (stdlib, ablation table parser + recipe picker)

Prerequisites: Phase 12 · 05 (LLaVA baseline)

#Learning Objectives

  • Name the five-axis VLM design space: image encoder, connector, LLM, data mix, resolution schedule.
  • Read an MM1 / Idefics2 / Cambrian-1 ablation table and predict which knob moves a given benchmark.
  • Pick a recipe (encoder, connector, data, resolution) for a new VLM given a compute budget and task mix.
  • Explain why detailed human captions beat GPT-4V distillation at the same token count.
Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.