Events, Not Pixels: Where World Models Get the Return Wrong

2026-10-23 12:00 arXiv cs.AI Haoyue Liu, Xiaoying Tang 约 11 分钟读完

World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful rollout can still assign the wrong return. To identify where this mismatc…

Authors: Haoyue Liu , Xiaoying Tang

View a PDF of the paper titled Events, Not Pixels: Where World Models Get the Return Wrong, by Haoyue Liu and 1 other authors

View PDF

HTML (experimental)

Abstract: World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful rollout can still assign the wrong return. To identify where this mismatch arises, we introduce a transition-level diagnostic that replaces imagined transitions with real ones and measures the reduction in return error against a budget-matched random control. Our analysis yields two findings. (1) Event transitions, the few steps where a reward is earned or an episode ends, are rendered as well as other steps (median pixel-error ratio 0.85), yet restoring them removes a median of 70% of the return error. In contrast, restoring transitions with the largest MSE, SSIM or LPIPS error removes less than a random choice. Across the 44 Atari-100k model--collection pairs that pass our reliability checks, including three released families, event patches outperform random patches in 39 (sign test p=1.4 x 10^{-7}), with a median gain of +44 percentage points. (2) The remaining error is structured: a complementary patch separates errors at true events from those elsewhere, while an eight-step timing tolerance recovers few events missed by our deterministic models. This structure also guides repair: a rule recorded before four repair outcomes were known predicts when a diffusion model repairs these failures. Event patches improve plan ranking, and the same concentration appears on control tasks with rare reward events.

Subjects:

Artificial Intelligence (cs.AI)

Cite as:

arXiv:2610.29435 [cs.AI]

(or

arXiv:2610.29435v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.29435

Focus to learn more

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Haoyue Liu [ view email ]

[v1]

Wed, 21 Oct 2026 08:59:02 UTC (1,087 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Events, Not Pixels: Where World Models Get the Return Wrong, by Haoyue Liu and 1 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.AI

< prev

next >

new

recent

| 2026-10

Change to browse by:

References & Citations

SciX

Google Scholar

Semantic Scholar

export BibTeX citation

BibTeX formatted citation

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer ( What is the Explorer? )

Connected Papers Toggle

Connected Papers ( What is Connected Papers? )

Litmaps Toggle

Litmaps ( What is Litmaps? )

scite.ai Toggle

scite Smart Citations ( What are Smart Citations? )

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv ( What is alphaXiv? )

Links to Code Toggle

CatalyzeX Code Finder for Papers ( What is CatalyzeX? )

DagsHub Toggle

DagsHub ( What is DagsHub? )

GotitPub Toggle

Gotit.pub ( What is GotitPub? )

Huggingface Toggle

Hugging Face ( What is Huggingface? )

ScienceCast Toggle

ScienceCast ( What is ScienceCast? )

Demos

Replicate Toggle

Replicate ( What is Replicate? )

Spaces Toggle

Hugging Face Spaces ( What is Spaces? )

Spaces Toggle

TXYZ.AI ( What is TXYZ.AI? )