activevisionai/ActiveVision-Results

ActiveVision Results

Model predictions and per-task scores for the ActiveVision benchmark: every final answer behind the leaderboard. 58 chain-of-thought configurations and 9 tool-using coding agents, each on all 85 items (17 tasks × 5).

🌐 Per-task results page · 🖼️ Benchmark dataset · 📄 Paper · 💻 Evaluation code

from datasets import load_dataset
preds = load_dataset("activevisionai/ActiveVision-Results", "predictions", split="train")  # 5,695 rows
board = load_dataset("activevisionai/ActiveVision-Results", "leaderboard", split="train")  # 68 rows

# join with the benchmark images and questions on `id`
items = load_dataset("activevisionai/ActiveVision", split="train")

predictions

One row per (configuration, item).

Field	Description
id, task, category	the benchmark item (same values as in activevisionai/ActiveVision)
track	cot (pure chain-of-thought, no tools) or agent (tool-using coding agent)
model	model id, e.g. gpt-6-astra, claude-opus-5-5
harness	none for chain-of-thought; codex or claude-code for agents
reasoning_effort	effort / thinking level the model was run at
input	image; text-only marks a no-image control run
prediction	the extracted final answer (empty if the model gave no parsable answer)
answer	ground truth
correct	graded with the rule below

leaderboard

One row per configuration plus the human reference (track = human): overall correct out of total, accuracy, cost_per_item_usd (API list price; empty for humans), and one column per task with the number of items correct out of 5. The human row is the mean of 3 unaided participants.

Grading

The prediction is the contents of the model's last <answer>...</answer> block, compared by exact match after normalizing case, whitespace, and separators ("C, A, G" ≡ "CAG"; integer answers also match numerically, "07" ≡ 7). See scoring.py.

Only final answers are released — no reasoning text and no agent tool-call traces.

Citation

@article{zhang2026exam,
  title={An exam for active observers},
  author={Zhang, Jiarui and Tao, Muzi and Wang, Shangshang and Liu, Ollie and Ma, Xuezhe and Neiswanger, Willie},
  journal={arXiv preprint arXiv:2607.16165},
  year={2026}
}