Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
Abstract
An LLM-based pipeline audits simulated manipulation success checkers by recreating disagreement scenarios and visually verifying whether observed behavior satisfies task instructions, revealing phantom successes and silent failures.
Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.
Community
Are your robot policies really succeeding, or is the success checker lying? ๐ค
Simulated manipulation benchmarks rely on hand-written success checkers, which are usually validated only against expert demos. Learned policies behave differently, so checkers can:
- โ accept incomplete executions โ phantom success
- โ reject executions that actually follow the instruction โ silent failure
We introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. It reads the instruction and checker code, proposes states where they may disagree, recreates them in simulation, and verifies the outcome with multi-view renderings.
๐ Key finding: checker errors appear across popular benchmarks (e.g., RoboTwin 2.0 errs in both directions), so reported success rates may not mean what we think.
Feedback welcome, especially from people building or using manipulation benchmarks!
The idea of auditing the checker rather than the policy is what stuck with me, because a reported success rate is only as good as the rule that labels it. Finding 40 phantom successes next to 21 silent failures shows a checker validated only on expert demos can be wrong in both directions. Did the visual verification ever overturn the checker in a case where the checker turned out to be right?
Thanks for the insightful question! We do not treat the VLM-based multi-view verification as ground truth, but rather as a filtering mechanism for identifying potential checker defects. All 61 reported defects were independently verified by two human annotators. In our human validation, we did not observe any cases where visual verification incorrectly overturned a correct checker verdict. That said, we do not claim that VLM-based verification is infallible.
Get this paper in your agent:
hf papers read 2611.25179 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper