An image is a 2-D tensor. A video is a 3-D one. The theory is the same; the compute is 10-100x harder. OpenAI's Sora (Feb 2024) proved it was possible. By 2026 Veo 2, Kling 1.5, Runway Gen-3, Pika 2.0, and WAN 2.2 ship production video from text at 1080p — and the open-weights stack (CogVideoX, HunyuanVideo, Mochi-1, WAN 2.2) is 12 months behind.
A 10-second 1080p video at 24fps is 240 frames of 1920×1080×3 pixels. That's ~1.5 GB of raw data per clip. Pixel-space diffusion is infeasible. You need:
Spatiotemporal compression. A VAE that encodes videos, not frames, into a sequence of spatial-temporal patches.
Temporal coherence. Frames need to share content, lighting, and object identity over seconds. The net has to model motion.
Compute budget. Video training is 10-100x more expensive than image for the same model size.
Conditioning. Text, image (first-frame), audio, or another video. Most production models accept all four.
The architecture that solved this is the Diffusion Transformer (DiT) applied to spatiotemporal patches, trained on huge (prompt, caption, video) datasets. Same diffusion loss as Lesson 06.
Full Access lesson
Keep going. The rest of this lesson is one step away.
Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.