Papers
arxiv:2611.25179

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

Published on Nov 18
ยท Submitted by
Zirui Song
on Nov 20
Authors:
,
,
,
,
,

Abstract

An LLM-based pipeline audits simulated manipulation success checkers by recreating disagreement scenarios and visually verifying whether observed behavior satisfies task instructions, revealing phantom successes and silent failures.

Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.

Community

Paper submitter

Are your robot policies really succeeding, or is the success checker lying? ๐Ÿค–

Simulated manipulation benchmarks rely on hand-written success checkers, which are usually validated only against expert demos. Learned policies behave differently, so checkers can:

  • โœ… accept incomplete executions โ†’ phantom success
  • โŒ reject executions that actually follow the instruction โ†’ silent failure

We introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. It reads the instruction and checker code, proposes states where they may disagree, recreates them in simulation, and verifies the outcome with multi-view renderings.

๐Ÿ” Key finding: checker errors appear across popular benchmarks (e.g., RoboTwin 2.0 errs in both directions), so reported success rates may not mean what we think.

Feedback welcome, especially from people building or using manipulation benchmarks!

The idea of auditing the checker rather than the policy is what stuck with me, because a reported success rate is only as good as the rule that labels it. Finding 40 phantom successes next to 21 silent failures shows a checker validated only on expert demos can be wrong in both directions. Did the visual verification ever overturn the checker in a case where the checker turned out to be right?

ยท

Thanks for the insightful question! We do not treat the VLM-based multi-view verification as ground truth, but rather as a filtering mechanism for identifying potential checker defects. All 61 reported defects were independently verified by two human annotators. In our human validation, we did not observe any cases where visual verification incorrectly overturned a correct checker verdict. That said, we do not claim that VLM-based verification is infallible.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.25179
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.25179 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.25179 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.25179 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.