Sign inStart free
AI Engineering/Phase 12/Lesson 22

Document and Diagram Understanding

Documents are not photos. A PDF, scientific paper, invoice, or handwritten form has layout, tables, diagrams, footnotes, headers, and semantic structure that plain image understanding cannot capture. The pre-VLM stack was a pipeline: Tesseract OCR + LayoutLMv3 + table-extraction heuristics. The VLM wave replaced that with OCR-free models — Donut (2022), Nougat (2023), DocLLM (2023) — that emit structured markup directly. By 2026 the frontier is just "feed the page image to Claude Opus 4.7 at 2576px native," and the structured-markup output comes for free. This lesson reads the three-era arc of document AI.

Build 3 hr Python (stdlib, layout-aware document parser skeleton)

Prerequisites: Phase 12 · 05 (LLaVA), Phase 5 (NLP)

#Learning Objectives

  • Explain the three eras of document AI: OCR pipeline, OCR-free, VLM-native.
  • Describe LayoutLMv3's three input streams: text, layout (bbox), image patches, with unified masking.
  • Compare Donut (OCR-free, image → markup), Nougat (scientific paper → LaTeX), DocLLM (layout-aware generative), PaliGemma 2 (VLM-native).
  • Pick a document model for a new task (invoices, scientific papers, handwritten forms, Chinese receipts).
Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.