AI Engineering/Phase 04/Lesson 28

World Models & Video Diffusion

A video model that predicts the next seconds of a scene is a world simulator. Condition that prediction on actions and you have a learned game engine.

Learn + Build 1.3 hr Python

Prerequisites: Phase 4 Lesson 10 (Diffusion), Phase 4 Lesson 12 (Video Understanding), Phase 4 Lesson 23 (DiT + Rectified Flow)

#Learning Objectives

Explain the difference between a pure video generation model (Sora 2) and an action-conditioned world model (Genie 3, DreamerV3)
Describe a video DiT: spatio-temporal patches, 3D position encoding, joint attention across (T, H, W) tokens
Trace how a world model plugs into robotics: VLM plans → video model simulates → inverse dynamics emits actions
Pick between Sora 2, Genie 3, Runway GWM-1 Worlds, Wan-Video, and HunyuanVideo for a given use case (creative video, interactive sim, autonomous-driving synthesis)

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.