AI Engineering/Phase 14/Lesson 20

Benchmarks: WebArena and OSWorld

WebArena tests web-agent capability across four self-hosted apps. OSWorld tests desktop-agent capability across Ubuntu, Windows, macOS. At release (2023–2024) both showed a big gap between best-in-class agents and humans. The gap is narrowing; the failure modes haven't changed.

Learn 1 hr Python (stdlib)

Prerequisites: Phase 14 · 19 (SWE-bench, GAIA)

#Learning Objectives

Describe WebArena's four self-hosted apps and why execution-based evaluation matters.
Explain why OSWorld uses real OS screenshots instead of accessibility APIs.
Name the two primary OSWorld failure modes: GUI grounding and operational knowledge.
Summarize what OSWorld-G and OSWorld-Human add on top of the base benchmark.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.