Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

GPT-6 Astra and Claude Fable 5.1 barely pass!?

Today's LLM agents are strong when the problem is already well posed — write code, solve a math problem, call an API. The real world never hands you a problem that way. Nobody defines your variables, tells you which model to build, where the constraints are, how dirty the data is, or whether the conclusion survives contact with reality.

Compressing a messy real-world situation into a model that is computable, verifiable, and defensible — that is mathematical modeling. It is the interface layer an LLM agent must cross to go from solving exercises to doing real work, and it is the least adequately measured layer we have.

TB-MathModel is evaluation infrastructure built for exactly that layer.

We use CUMCM/MCM contest problems as the entry point, and not because we care about contests: they are one of the very few public corpora that bundle a complete, unstructured real-world problem with genuine attachments, official figures, and human expert solutions. Eighteen of them go straight into containerized terminals — read the statement, open the spreadsheets, look at the figures, write code, run the solver, emit one structured JSON. That is a miniature of real modeling work.

Three design choices make this layer measurable:

1. We do not grade the report — we check the numbers. Each task anchors on the numeric core of the final question, scored direction-aware and one-sided: beating the human paper never costs you points.

2. Not one reference answer — 94 of them. Modeling has no unique solution, so every Outstanding-award paper gets a vote: a model has to land close to many human experts at once, not memorize one answer.

3. Feasibility verified independently, and machine-checkable. A programmatic gate over six decidable invariant forms plus 624 kernel-checked Lean 4 obligations — no Mathlib, no sorry. Reporting a number is not enough; it has to survive the physics.

The result: the best entry, GPT-6 Astra, reaches only 64.6%; Fable 5.1 reaches 63.1%; a $2.54 open model holds 59.8% at 1/100 the cost.

We are still far from agents that can handle real-world problems. And what that road needs is a uniform environment, a verifiable reward, and an agent-trajectory corpus that compounds — precisely what TB-MathModel sets out to provide.

Code and data: https://github.com/wxj630/terminal-bench-math-modeling