LocalLLaMA/typed-decisions

Typed Decisions

A benchmark for typed probabilistic decisions. A model gets one piece of unstructured state and answers five typed questions about it at once, and every answer is a probability distribution, not a single label.

The schema follows the System One primitives (noul, choice, score) used by TypeSafe AI, so a row replays against any API with that shape. The benchmark is independent: it is not affiliated with TypeSafe and does not reproduce their Jev model.

Leaderboard

test split: 400 cases, 2,000 decisions. General models are scored zero-shot; they have never seen these workflows or their question schemas.

#	Model	Kind	Accuracy ↑	KL from gold ↓	Brier ↓	ECE ↓	p50 latency	Price / 1M input
1	meraGPT Decider 1 (sd-1)	general, zero-shot	0.768	0.096	0.052	0.180	526 ms	$0.03
2	Liquid AI d1 (d1:free)	general, zero-shot	0.742	0.475	0.155	0.124	525 ms	free tier
3	TypeSafe Jev 1.13.0	general, zero-shot	0.727	1.442	0.148	0.144	710 ms	$0.042
4	Featherless Simple Jev (Qwen3.6-35B-A3B-classifier)	general, zero-shot	0.716	0.488	0.176	–	–	free demo
5	ModernBERT-base (149M)	specialist, fitted per workflow	0.646	0.223	0.119	0.179	349 ms†	–
6	MiniLM-L6 (22M)	specialist, fitted per workflow	0.587	0.262	0.143	0.108	22 ms†	–
7	Jeff-Gemma4-E2B	general, zero-shot, open weights	0.561	0.403	0.219	0.188	2,272 ms†	open weights
8	Jeff-Qwen3.5-2B	general, zero-shot, open weights	0.511	0.460	0.237	0.203	1,346 ms†	open weights
9	Jeff-Qwen3.5-0.8B	general, zero-shot, open weights	0.483	0.679	0.313	0.251	662 ms†	open weights
–	Prior (ignores the input)	reference	0.470	0.347	0.189	0.088	–	–
–	Uniform (same probability on every option)	reference	0.308	0.444	0.238	0.169	–	–

Hosted rows are p50 end to end from a client, one request at a time. † Measured on the same machine as the model (M3 Max), not comparable with hosted latency.

meraGPT Decider 1 is state of the art on this benchmark. It leads Jev on every question type (noul 0.840 vs 0.775, choice 0.733 vs 0.720, score 0.739 vs 0.696), its distributions sit far closer to the gold (KL 0.096 vs 1.442), it is faster end to end, and it costs less per token. It answers POST /v1/systemone at meragpt.com, so the typesafe-sdk works against it by setting TYPESAFE_BASE_URL.

To add a model, score it on test with the full distributions and open a discussion with the numbers and the mode (specialist or general) it used.

Notes on the rows

• Liquid AI d1 was measured on 2026-09-30 through Liquid's API (https://api.liquid.ai/v1/systemone, model: d1:free), with the same client code as the Jev row: all 2,000 decisions, zero errors. It ties Decider 1 on noul (0.840) and choice (0.732 vs 0.733) and trails on score (0.677 vs 0.739), and it beats Jev on accuracy and KL. Liquid lists no per-token price yet, so the row shows the free tier.
• Jev 1.13.0 was measured on 2026-09-18 through TypeSafe's API (jev-latest, which reported jev-1.13.0): all 2,000 decisions, zero errors, $0.016 in total. Its accuracy is near the 0.735 ceiling, but it puts nearly all its probability on one answer, which is where the KL gap comes from. Its confidence is not badly calibrated (overconfidence +0.023); the gold is a three-sample spread it does not reproduce.
• Jeff (firelex/jeff, Apache-2.0) was scored on 2026-09-29 through the same client code as Jev, against Jeff's own jeff-serve (commit 2c1bfce) with the calibration each checkpoint ships. All three clear the Prior on accuracy but not on KL. Jeff's own 83.1% comes from a different five-benchmark panel. If there is a better way to serve them, open a discussion and we will rescore.
• Specialists use Adaptive Classifier 0.2.0, one classifier per question on a frozen encoder, tuned on a held-out quarter of train (mean pooling, max_length 512, 30 epochs, prototype_weight 0.3). Each case is entered four times, split across labels in proportion to its gold, so the soft target survives hard-label training; that cut KL by a third.
• Prior answers each question's train label frequencies for every case. It has the best ECE while knowing nothing, which is why KL and Brier are the columns to read, not ECE.

Reading a score

Gold is the mean of three samples from a teacher of roughly 4B-class capability, so a score measures agreement with that teacher, not correctness. A better model can score lower wherever the teacher is wrong (it missed a duplicate invoice whose ID matched an earlier one).

Reference	Accuracy	What it is
Prior	0.470	the floor: label frequencies, ignoring the input
Perfect factor recovery	0.704	a model fitted to the latent factors that generated each case
Teacher self-agreement	0.735	a fresh teacher sample against gold built from the others

Scores well above 0.735 mean a model is learning the teacher's quirks. Per-question ceilings vary from 0.560 (agent_trace/urgency) to 0.937 (customer_service/category), so read scores per question as well as on average.

Specialist and general numbers are not comparable. A specialist is fitted on train for these four workflows and cannot answer anything else. A general model takes any question schema at request time and has never seen these. Say which mode you used; the gap between them is the price of generality, not a quality ranking.

The data

Type	Answer	Shape
noul	yes/no	probability that the statement is true
choice	one of N labels	distribution over labels, plus confidence
score	an ordered rubric	distribution over levels, plus an expected score

Every option carries a written description in criteria, and the descriptions are part of the input.

Workflow	Decision	Train	Test
agent_trace_observability	Does an agent run need human review, and how urgently?	300	100
customer_service	The right response and action for a customer thread and account.	300	100
invoice_processing	Pay, hold or reject a vendor bill against its order and delivery.	300	100
security_incidents	Close, investigate or contain a security alert, given machine history.	300	100

state and questions together are exactly the body of a POST /v1/systemone request. gold holds the full gold distributions, and flat <question>__label / __probabilities / __score / __probability_true columns hold the same answers for convenience. factors and label_agreement describe how the case was built and are not model input.

from datasets import load_dataset
import json

ds = load_dataset("LocalLLaMA/typed-decisions", "customer_service", split="test")
row = ds[0]
state, questions, gold = (json.loads(row[k]) for k in ("state", "questions", "gold"))
print(gold["urgency"]["probabilities"])   # score against the full distribution

Report KL or log loss and Brier next to accuracy; calibration is the point.

How it was built

Each case starts from independently sampled latent factors (topic, tone, severity, discrepancy type and so on), which a model renders into free text where the artefact is textual and keeps structured where it is structured. A teacher labels each case three times at temperature 0.7, and the gold is the mean of those distributions, so it stays soft where a decision is genuinely ambiguous. Before release, an audit checks state diversity, label balance, and that the gold actually tracks the input. train comes from a separate run at a different seed; packaging refuses to build if any case id or state appears in both splits.

The write-up behind the benchmark, including the two bugs it exposed in the classifier library: Typed Decisions on Latent Node.