ScienceVision

ScienceVision is a benchmark of 250 expert-authored questions about 3D scientific data. Each task pairs an English question with a self-contained asset bundle — a 3D scene, a molecular structure, a CAD assembly, or a medical volume — and asks the model to read a quantity, identify a component, or make a geometric decision from that data.

Read the ScienceVision blog post for the release overview.

The benchmark covers five domains, 50 tasks each: architecture, biology, chemistry, mechanical CAD and medical imaging. Tasks are grouped into 22 sub_domain buckets and 30 direction labels.

What each task contains

Every task is a self-contained directory:

<domain>/<task_id>/
  question.md                  the English question
  answer.json                  the graded gold and its shape
  normalized.json              the canonical question, answer and taxonomy
  task_and_trajectory.json     the expert's annotated trajectory
  _meta.json                   task metadata and step/asset counts
  allow_mcp.json               the MCP tools the task may use
  assets/                      the 3D or volume assets the question is about
  images/                      expert screenshots referenced by the trajectory
  views/                       rendered 2D views (four view settings)
  views-aligned/               aligned 2D views (four view settings)

A verifier-side rubric lives outside the task directory at verifier/scoring/<task_id>.json. It is not staged into the agent environment.

Every task also ships a reference solution under oracles/<task_id>/solution/ (solution.sh, oracle.py, reasoning.md, oracle_trajectory.jsonl, notes.json). The oracle drives the ScienceVision MCP tools inside the task container, writes /logs/agent/sci3d_answer.json, and scores 1.0 through the task's own scorer.

Distribution

Domain	Tasks	sub_domain (tasks)
architecture	50	arch_alignment_and_stacking 12 · arch_dimensional_compliance 16 · arch_interference_and_clearance 12 · arch_object_geometry_readings 10
biology	50	bio_interaction_typing 8 · bio_molecular_contact 14 · bio_occluded_target_geometry 6 · bio_proximity_choice 8 · bio_spatial_arrangement 14
chemistry	50	chem_conformation_occlusion 21 · chem_crystal_periodicity 4 · chem_dihedral_measurement 9 · chem_hybridization_state 7 · chem_molecular_extent 9
mechanical-cad	50	cad_assembly_composition_count 10 · cad_mechanism_motion 6 · cad_part_instance_count 20 · cad_part_property_identification 14
medical	50	med_anatomy_pointing 5 · med_geometry_measurement 8 · med_lesion_pointing 25 · med_slice_comparison 12

Every answer is one JSON object: 143 tasks ask for a single field, 107 for two or more (a reading plus the evidence it rests on).

Domain	assets/ files	Formats (files)
architecture	57	GLB 35 · JSON 16 · PNG 6
biology	50	CIF 50
chemistry	50	XYZ 46 · CIF 4
mechanical-cad	110	PNG 41 · STEP 32 · STP 18 · SVG 11 · DXF 8
medical	58	NIfTI 56 · PNG 2

Usage

The tabular task table loads with the standard datasets API:

from datasets import load_dataset
import json

ds = load_dataset("UnipatAI/ScienceVision", split="test")
row = ds[0]
question = row["question"]
answer = json.loads(row["answer_json"])

Each row carries public_id, task_id, domain, sub_domain, direction, the full question, the gold in answer_json, the declared answer_fields, step and asset counts, the asset list, and the relative paths of the task and oracle directories.

To use an asset, download the repository and resolve the path relative to the row's task_dir:

from huggingface_hub import snapshot_download

root = snapshot_download("UnipatAI/ScienceVision", repo_type="dataset")
task_dir = f"{root}/{row['task_dir']}"

ID_MAP.csv maps the stable public_id to the original task_id. MANIFEST.json records the release counts.

Scoring

Task answers are graded by a deterministic scorer. The verifier-side rubric is in verifier/scoring/<task_id>.json; merge its scorer block into the task record at evaluation time. Tasks without a file use the default structured-answer scorer.

Harbor tasks

harbor/ contains a generator and one manifest per task. It produces runnable Harbor task directories without duplicating the asset tree in this repository:

python harbor/build_tasks.py --out /tmp/sciencevision-harbor --route cc-mcp

The generated tasks use the standard Harbor layout (task.toml, instruction.md, tests/, environment/) and stage the task's assets or rendered views into the per-task Docker build context. See harbor/README.md for the available routes, the manifest schema, and the image prerequisite.

Quality

The release was checked task by task:

• 250/250 tasks have a gold answer; answer.json, normalized.json and task_and_trajectory.json agree in all 250.
• 250/250 golds score themselves under their rubric; 250/250 oracle answers score 1.0.
• 250/250 tasks carry their expert steps; 2,405 of 2,406 steps have their screenshot.
• 0 non-ASCII file names, 0 text files with Chinese, 0 internal paths.

Remaining known issues are listed in KNOWN_ISSUES.md.

License

Released under the MIT License; see LICENSE.

Citation

If you use ScienceVision, cite the release:

@misc{sciencevision2026,
  title        = {ScienceVision: Can AI Observe and Reason Like a Scientist?},
  author       = {{Unipat AI}},
  year         = {2026},
  howpublished = {\url{https://unipat.ai/blog/ScienceVision}},
  note         = {Dataset: \url{https://huggingface.co/datasets/UnipatAI/ScienceVision}}
}