The dataset viewer is not available for this split.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
schema_version: int64
key: string
task_id: string
trial: int64
benchmark: string
model: string
started_at: string
task_metadata: struct<competition: string, problem_idx: string, problem_type: list<item: string>, source: double, s (... 41 chars omitted)
child 0, competition: string
child 1, problem_idx: string
child 2, problem_type: list<item: string>
child 0, item: string
child 3, source: double
child 4, strict_parsing: bool
child 5, tools_allowed: bool
prompt_hash: string
generation_hash: string
status: string
response: string
response_metadata: struct<usage: struct<prompt_tokens: int64, total_tokens: int64, completion_tokens: int64, prompt_tok (... 220 chars omitted)
child 0, usage: struct<prompt_tokens: int64, total_tokens: int64, completion_tokens: int64, prompt_tokens_details: n (... 29 chars omitted)
child 0, prompt_tokens: int64
child 1, total_tokens: int64
child 2, completion_tokens: int64
child 3, prompt_tokens_details: null
child 4, reasoning_tokens: int64
child 1, latency_seconds: double
child 2, id: string
child 3, model: string
child 4, finish_reason: string
child 5, endpoint: string
child 6, dispatcher_queue_wait_seconds: double
child 7, attempt: int64
child 8, request_token_budget: null
trajectory: list<item: struct<role: string, content: string, reasoning_content: string, tool_calls: null>>
child 0, item: struct<role: string, content: string, reasoning_content: string, tool_calls: null>
child 0, role: string
child 1, content: string
child 2, reasoning_content: string
child 3, tool_calls: null
finished_at: string
scoring_started_at: string
scoring_hash: string
score: struct<value: double, correct: bool, metrics: struct<answer_extracted: bool, predicted_answer: strin (... 106 chars omitted)
child 0, value: double
child 1, correct: bool
child 2, metrics: struct<answer_extracted: bool, predicted_answer: string, reference_answer: string, strict_parsing: b (... 40 chars omitted)
child 0, answer_extracted: bool
child 1, predicted_answer: string
child 2, reference_answer: string
child 3, strict_parsing: bool
child 4, parse_mode: string
child 5, scorer: string
child 3, explanation: null
model.yaml: string
suite.yaml: string
to
{'model.yaml': Value('string'), 'suite.yaml': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
schema_version: int64
key: string
task_id: string
trial: int64
benchmark: string
model: string
started_at: string
task_metadata: struct<competition: string, problem_idx: string, problem_type: list<item: string>, source: double, s (... 41 chars omitted)
child 0, competition: string
child 1, problem_idx: string
child 2, problem_type: list<item: string>
child 0, item: string
child 3, source: double
child 4, strict_parsing: bool
child 5, tools_allowed: bool
prompt_hash: string
generation_hash: string
status: string
response: string
response_metadata: struct<usage: struct<prompt_tokens: int64, total_tokens: int64, completion_tokens: int64, prompt_tok (... 220 chars omitted)
child 0, usage: struct<prompt_tokens: int64, total_tokens: int64, completion_tokens: int64, prompt_tokens_details: n (... 29 chars omitted)
child 0, prompt_tokens: int64
child 1, total_tokens: int64
child 2, completion_tokens: int64
child 3, prompt_tokens_details: null
child 4, reasoning_tokens: int64
child 1, latency_seconds: double
child 2, id: string
child 3, model: string
child 4, finish_reason: string
child 5, endpoint: string
child 6, dispatcher_queue_wait_seconds: double
child 7, attempt: int64
child 8, request_token_budget: null
trajectory: list<item: struct<role: string, content: string, reasoning_content: string, tool_calls: null>>
child 0, item: struct<role: string, content: string, reasoning_content: string, tool_calls: null>
child 0, role: string
child 1, content: string
child 2, reasoning_content: string
child 3, tool_calls: null
finished_at: string
scoring_started_at: string
scoring_hash: string
score: struct<value: double, correct: bool, metrics: struct<answer_extracted: bool, predicted_answer: strin (... 106 chars omitted)
child 0, value: double
child 1, correct: bool
child 2, metrics: struct<answer_extracted: bool, predicted_answer: string, reference_answer: string, strict_parsing: b (... 40 chars omitted)
child 0, answer_extracted: bool
child 1, predicted_answer: string
child 2, reference_answer: string
child 3, strict_parsing: bool
child 4, parse_mode: string
child 5, scorer: string
child 3, explanation: null
model.yaml: string
suite.yaml: string
to
{'model.yaml': Value('string'), 'suite.yaml': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Qwen3.5-9B IT agentic update (2026-11-13): Latest aligned rerun: SciCoding-Agent 20/100 (20.00%); TB 2.1 38/178 (21.35%), with two unscored verdicts retained as null. H100, TP2, temperature 0.7, top-p 0.95, top-k 20, two tasks per engine. Reasoning results are unchanged.
Latest SFT agentic update (2026-11-13): SciCoding 37/100; TB 49/178 with one native verdict missing. These exports use temperature 0.7, top-p 0.95, top-k 20 and supersede earlier SFT agentic scores/configs below. Reasoning results are unchanged.
Step-180 Research/HLE update — 2026-11-12
Canonical step-180 scores are now Research 35.70% mean rubric / 15.56% trajectory ≥7, and HLE 33.88%, from the fully scored Nebius H100 rerun. Earlier step-180 Research/HLE values below are historical. arXiv avg@20, agentic benchmarks and other checkpoints are unchanged.
arXiv Math avg@20 publication — 2026-11-12
The following 13 model/checkpoint folders now contain the complete avg@20 arXiv results and trajectories. Each has 103 questions × 20 trials = 2,060 scored responses, with no unresolved scores. These replace the older canonical arXiv evaluations; other benchmarks and unlisted checkpoints are unchanged.
| Model/checkpoint | Correct / 2,060 | avg@20 |
|---|---|---|
| Qwen3.5-35B-A3B | 737 | 35.78% |
| Gemma-4-12B-it | 649 | 31.50% |
| rl-stage1-mix/step-180 | 640 | 31.07% |
| rl-stage1-mix/step-120 | 620 | 30.10% |
| rl-agentic-step60 | 607 | 29.47% |
| Nemotron-Cascade-2-30B-A3B | 584 | 28.35% |
| Qwen3.5-9B | 576 | 27.96% |
| granite-4.2-8b | 549 | 26.65% |
| sft-mix | 545 | 26.46% |
| sft-agentic-stage2 | 533 | 25.87% |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 527 | 25.58% |
| gpt-oss-20b | 426 | 20.68% |
| Olmo-3-32B-Think | 404 | 19.61% |
OLMo uses native 64K context / 60K output; the other runs use 96K output. Model-specific profiles and scorer recovery/adjudication provenance are included. Older arXiv values below are historical where superseded by this table.
Previous repository documentation
Comparison-scope update: Nemotron Bolt E4B is archived under backup/NVIDIA-Nemotron-3.5-Bolt-E4B/ and excluded from report comparisons. Its original scores and trajectories are retained.
Latest publication — 2026-11-08
Finished evaluation exports include complete generated trajectories, original judgments/native verifier artifacts, frozen configurations and SHA-256 manifests. Scores with unresolved judgments remain provisional; native verifier success and normal agent completion are distinct.
| Newly published model | SciCoding-Agent avg@2 | TB 2.1 avg@2 | Caveat |
|---|---|---|---|
| Granite 4.2 8B | 40/100 (40%) | 32/178 (17.98%) | FS Research 1,799/1,800 scored; SciCoding repeat pending |
| Nemotron Cascade 2 30B A3B | 33/100 (33%) | 45/178 (25.28%) | 2 unscored TB attempts |
| SFT agentic stage 2 — November 13 rerun | 37/100 (37%) | 49/178 (27.53%) | TB has one unscored attempt; temperature 0.7 |
The four external-model suites include both FrontierScience tracks, arXiv Math and combined HLE-Verified Gold + Revision. Muse Glimmer's suite is archived under backup/muse-glimmer-30b/, with 64/100 SciCoding and 69/178 TB (2 unscored; one separately documented user-adjudicated zero).
Report comparison exclusions
DeepSeek Flash, GPT-OSS-120B, Muse Glimmer, GPT-5.6 Luna and Nemotron Bolt E4B are under backup/, not in the report comparison cohort. No trajectories were discarded during this move. Earlier revisions retain the former layout and superseded scores.
Protocol caveat
This publication does not rerun or rescore any model. External-model runs retain their approved original frozen runtime. The new SFT SciCoding export uses budget policy full-remaining-budget-no-tail-retry-v11; prior exports do not all share this policy. Consult each run's configs/provenance, rather than assuming identical budget handling or runtime solely from aligned sampling.
Earlier publication notes (superseded where updated above)
Science evaluation results
Report comparison scope
DeepSeek Flash, GPT-OSS-120B, Muse Glimmer, GPT-5.6 Luna and Nemotron Bolt E4B are archived under backup/ and excluded from the report comparison cohort. Their data are preserved, not discarded. Other model folders remain at the repository root.
Current aligned agentic reruns — published 2026-11-08
These eight benchmark folders supersede their older exports and any historical numbers below. Full trajectories, verifier outputs, native artifacts and effective native configurations are included. No new generation or scoring was performed for publication.
| Model | SciCoding-Agent avg@2 | Terminal-Bench 2.1 avg@2 |
|---|---|---|
| sft-agentic-stage2 | 37/100 (37.00%) | 49/178 (27.53%); 1 unscored |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 19/100 (19.00%) | 35/178 (19.66%) |
| Qwen3.5-35B-A3B | 41/100 (41.00%) | 62/178 (34.83%) |
| gpt-oss-20b | 22/100 (22.00%) | 12/178 (6.74%) |
All 1,112 attempts are scored: 1,103 native scores and nine explicitly adjudicated timeout zeros. The metric remains verifier pass; the proposed additional normal-completion requirement is NOT applied here.
The reruns use H100, Terminus 2, temperature 1.0, top-p 0.95, top-k 20, concurrency 2, avg@2. TP is model-specific (GPT-OSS-20B TP1; others TP2). Read AGENTIC_CONFIG_ALIGNMENT.md for remaining historical execution differences and the Luna exception. Do not claim that every historical run has an identical full configuration.
Previously published repository documentation
The following is retained for other benchmarks and provenance. For the four models' agentic results, use the current table and benchmark folders above.
Science evaluation results
Results are organized by model and benchmark. See each model README for protocol and completeness caveats.
DeepSeek Flash-0731 High (2026-10-28)
Results and full available trajectories are now included in the same per-benchmark format. FS Olympiad, ArXivMath, SciCoding-Agent and TB 2.1 are complete. Research is provisional (1,799/1,800 scored); SciCode-Verified is partial (259/287 scored steps). Exact settings, original exceptions and grading completeness are preserved.
Gemma-4-12B-it (2026-10-29)
Full results and trajectories: all six benchmark groups complete. TB 2.1 is 41/178 (23.03% avg@2), including one explicitly user-adjudicated zero following a verifier setup timeout. Native errors, recovery backups and historical sampling settings are preserved.
Qwen3.5-35B-A3B and Nemotron Lightning (2026-10-29)
Qwen non-agentic results and Nemotron full-suite results include full available trajectories, judging and native artifacts. Nemotron TB has 174 native scores plus four explicitly user-adjudicated zeros; original failures are archived. Qwen agentic evaluation is not part of this publication.
Qwen3.5-35B-A3B agentic suite completed (2026-10-29)
Full Qwen suite and trajectories now includes SciCoding-Agent 36/100 (36.00% avg@2) and TB 2.1 61/178 (34.27% avg@2). TB includes two explicit user-adjudicated zeros; original null native verdicts and timeout records are archived. Prior non-agentic results are unchanged.
GPT-OSS-120B High (2026-10-30)
Results and full available trajectories: FS Olympiad 63.50%, provisional Research rubric 41.04% (1799/1800 scored), arXiv Math 31.55%, SciCode main problems 59.38%. One Research judgment was API-policy-rejected and remains unscored, not zero. TB 2.1 and SciCoding-Agent have not started.
GPT-OSS-20B and OLMo-3-32B-Think (2026-10-30)
- GPT-OSS-20B: all requested reasoning and agentic trajectories, including the 69-attempt Harmony recovery. TB provisional 17/178 (9.55%) with two unscored attempts; SciCoding-Agent 22/100. Research retains one unscored judgment.
- OLMo-3-32B-Think: complete reasoning suite, including context-budget recoveries; arXiv Math 75/412 (18.20%), SciCode main problems 16/64 (25.00%). Native 64K context / up to 60K output.
See each model folder for complete benchmark summaries, frozen settings, full available trajectories and grading provenance.
HLE-Verified (2026-10-30)
HLE results are stored directly in each model folder under one combined hle-verified/ folder containing all 1,600 Gold + Revision text questions. Each model README and summary reports the combined accuracy; hle-verified/provenance.json preserves configuration, data/source hashes and recovery evidence. Subset labels remain in records for traceability.
| Model | Gold | Revision | Overall (1,600) |
|---|---|---|---|
| Qwen3.5-35B-A3B | 39.65% | 35.61% | 37.06% |
| sft-agentic-stage2 | 28.70% | 32.68% | 31.25% |
| gpt-oss-20b | 28.17% | 32.39% | 30.88% |
| Gemma-4-12B-it | 27.65% | 29.95% | 29.12% |
| Qwen3.5-9B | 29.57% | 28.20% | 28.69% |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | 23.30% | 28.49% | 26.62% |
| Olmo-3-32B-Think | 12.00% | 21.46% | 18.06% |
All 11,200 answers are scored. Gold has 575 text questions and Revision 1,025 per model. Overall is question-weighted accuracy. OLMo uses its native 64K context / up to 60K output; other models use 96K output. See the model-level protocol caveats.
- Downloads last month
- 790