LemonDecider 4B

Model ID: lemondecider-4b-20261023-v3
Base model: Qwen/Qwen3.5-4B

This v3 release makes typed decisions from a shared text state and named questions. It returns choice probabilities, yes/no probabilities (noul), or ordinal scores without generating an answer token stream. Requests that fit use one joint forward; overlength question lists are recursively bisected, preserving the complete state and every option in each chunk. Inputs are never truncated.

Model description

Property	Value
Architecture	Qwen3.5-4B hybrid backbone with a 255-option decision readout
Loader	modeling_lemondecider.LemonDeciderForDecisionClassification, trust_remote_code=True
Model type / protocol	lemondecider_decision / lemondecider-v1
Attention	Noncausal periodic full-attention layers; hybrid recurrent layers; SDPA
Readout	Native inline joint-question readout (joint_readout.predict_joint)
Temperature	1.0026751894886363, scalar calibrated
Capacity	8192 tokens per chunk including template; at most 255 options per question
Serving strategy	recursive-question-bisection-v1; short requests are not split
Precision	BF16 loading; model-specific readout precision preserved by packaged implementation
Serialized tensors	724
Modalities	Published HTTP service is text-only
Runtime	Python 3.12; evaluated runtime recorded Torch 2.9.0, Transformers 5.17.0; run notes specify FLA 0.5.2

Training

LemonDecider 4B v3 was fine-tuned (SFT) from a common Qwen3.5-4B initialization on 1,199,692 decision/QA rows (about 1.20M training rows, with separate development, calibration and test splits). The released weights are a uniform FP32 average of four checkpoints from the same SFT run, followed by scalar temperature calibration—not a combination of two independent training runs.

Question formats: choice (up to 255 options; 84.4%), noul (no/yes; 15.2%) and score (ordinal ratings; 0.4%). This training run is text-only, with the vision tower frozen. The original training corpus retains joint-question grouping where applicable; the single-record export preserves the question-level content but removes grouping metadata and is not an exact reproduction of the original joint-training layout.

Data categories:

• Service and action decisions — user-intent routing, case classification, agent-action safety and policy judgments;
• Language judgment — sentiment, sarcasm, natural language inference, factuality and paraphrase-style checks;
• Knowledge and reasoning QA — multiple-choice knowledge, commonsense and logical reasoning, arithmetic and math word problems;
• Retrieval and selection — relevance judgments, information retrieval and product selection;
• Preference and quality judgments — helpfulness, correctness, safety and response-quality comparisons;
• Probability and ordinal judgments — yes/no probabilities and ordered ratings.

Training parameters (from the recorded configuration and training log):

• Distributed FSDP with full sharding on 16 accelerators;
• One epoch, 4,690 optimization steps, effective batch size 256, learning rate 2e-6;
• Maximum sequence length and token budget 8,192; native joint-question inline readout;
• Objective recorded as option_cross_entropy; choice permutation global-reference; text-only inputs and frozen vision tower;
• Warmup ratio 0.15, weight decay 0.01, gradient clipping 1.0, gradient checkpointing enabled;
• Precision policy fp32-master-bf16, BF16 serving load, SDPA with noncausal full attention;
• Four-checkpoint weight averaging; final temperature 1.0026751894886363, fitted on 1,448 held-out calibration rows.

The corpus manifest records fingerprint-based filtering against protected JevBench and Decision Index evaluation inputs, while some source collections include benchmark training splits. This is not a blanket guarantee of zero overlap, nor a claim that every upstream source has permissive redistribution terms; source-specific terms still require review.

Package artifacts

Weight files

File	Bytes	SHA-256
model-00001-of-00002.safetensors	4,973,336,472	35c24b89f66f61dafba87f1249d98c9f038e2e4a3680dd93edc53a7689a8d783
model-00002-of-00002.safetensors	4,106,591,800	f7bef3b498174e1f5b13e357056b0ed25b3842466e91206a3157e151bcbc6139
tokenizer.json	19,989,492	6f32ce20dc35f57a7f9ad1eac03525bd7d30f9df8cea6507e958279cc3657706
model.safetensors.index.json	68,427	cdc57ae81f5d72a44461bdbb9949139319294b25ce35734646fb0c7b074f52f8

The repository keeps the model/processor/tokenizer filenames used by the HF layout. v3 additionally requires joint_readout.py; code/ provides the serving entry points, and requirements.txt describes installation dependencies. Original copyright and license notices are retained.

Evaluation results

1) Decision Index, edition 0.3 — FULL public suite (complete run)

On 2026-10-25, the chunked service performed a fresh full evaluation over 140,178 scoreable requests using four independent PPU-ZW810E workers. The 442 officially excluded requests were filtered before inference. The official scorer reported complete: true, with 140,148 ok and 30 unsupported.

Headline	Value
Decision Index (public 0.3)	60.03
Raw index	69.34
Breadth skill	58.63
Median / mean / p95 request latency	53.3 ms / 104.9 ms / 286.1 ms

This is a self-evaluated public-suite result, not an official private Full score or leaderboard acceptance. Latency is HTTP request wall time from a four-worker PPU run, not official single-GPU latency admission. Kit commit: 25c6ec7b6b5f44a216c63c3a700de3a9c6d6699d.

$0

Per-area scores

Area	Raw	Skill	Coverage	Benchmarks
Knowledge & Reasoning	0.5307	0.4044	0.9998	11
Language Understanding	0.7714	0.6795	1.0000	9
Retrieval & Classification	0.7635	0.6827	1.0000	6
Tools & Automation	0.7710	0.7431	0.9986	5
Arts & Human Taste	0.6299	0.4756	0.9979	6

Per-benchmark native scores (★ = included in the public index)

The Score column uses each benchmark's native summary metric on supported cases; Skill and Coverage come from the official index report. Track aggregation and coverage adjustment can therefore make Score differ from the index's raw value. iSarcasmEval uses its English track-A headline rather than a nonexistent combined summary score. All values below are generated from this v3 run, not the v2 card.

#	Benchmark	Metric	Requests	Score	Skill	Cov.
1	BFCL ★	case exact accuracy	1,694	0.9534	0.9370	1.0000
2	ToolRet ★	nDCG@10	685	0.6082	0.5455	0.9971
3	API-Bank ★	accuracy	508	0.8458	0.8394	0.9961
4	BANKING77 ★	macro-F1	3,080	0.9343	0.9335	1.0000
5	CLINC150+OOS ★	macro-F1	5,500	0.9296	0.9292	1.0000
6	RouterBench	selected quality (quality objective)	10,000	0.7992	0.5292	1.0000
9	Home appliance simulator ★	case exact accuracy	88	0.6023	0.6023	1.0000
10	SGD/SGD-X	macro-F1	2,500	0.4213	0.0371	1.0000
11	ContractNLI ★	macro-F1	123	0.8665	0.8070	1.0000
12	ANLI ★	macro-F1	3,200	0.7184	0.5782	1.0000
20	BPoMP ★	accuracy	5,000	0.9856	0.9705	1.0000
21	Humicroedit ★	accuracy	2,628	0.6355	0.2709	1.0000
22	POP909-CL ★	accuracy	2,000	0.1792	0.1687	0.9875
23	cfcolor ★	accuracy	5,000	0.7450	0.4409	1.0000
24	MMLU	accuracy	14,033	0.7959	0.7279	1.0000
25	GPQA Diamond ★	accuracy	196	0.4490	0.2653	1.0000
26	ARC-Easy	accuracy	2,376	0.9882	0.9843	1.0000
27	ARC-Challenge	accuracy	1,172	0.9565	0.9420	1.0000
28	WinoGrande ★	accuracy	1,267	0.8974	0.7948	1.0000
29	HellaSwag ★	accuracy	10,042	0.9552	0.9403	1.0000
30	GSM8K ★	accuracy	2,638	0.6274	0.5540	1.0000
31	ChessBench ★	accuracy	5,000	0.1304	0.0529	1.0000
32	MuSR ★	accuracy	752	0.5984	0.3615	1.0000
33	SATA-Bench ★	case exact accuracy	1,650	0.2921	0.2827	1.0000
34	SimpleBench	accuracy	10	0.0000	0.0000	1.0000
36	BRIGHT ★	nDCG@10	220	0.4594	0.3884	1.0000
37	Amazon ESCI ★	macro-F1	5,000	0.5954	0.4925	1.0000
38	ACOS ★	per-review F1	1,565	0.3822	0.3624	1.0000
39	FinEntity ★	macro-F1	979	0.9345	0.9037	1.0000
40	iSarcasmEval ★	Sarcasm F1 (track A, English)	4,600	0.6318	0.5263	1.0000
41	VAST ★	macro-F1	3,006	0.7751	0.6627	1.0000
42	NLI4CT ★	macro-F1	5,500	0.8090	0.6284	1.0000
43	CRUXEval ★	accuracy	570	0.6263	0.4071	1.0000
44	CLadder ★	accuracy	5,000	0.7908	0.5816	1.0000
45	HLE ★	accuracy	501	0.1080	0.0000	0.9980
50	Habermas Machine ★	accuracy	1,676	0.6193	0.4475	1.0000
56	PhishNChips phishing decisions ★	accuracy	2,000	0.7675	0.5350	1.0000
57	MMLU-Pro ★	accuracy	12,032	0.5865	0.5349	1.0000
58	BBH fixed-option tasks ★	accuracy	5,507	0.7118	0.5823	1.0000
59	RAGTruth response-level hallucination ★	F1 on hallucinated class	2,700	0.8433	0.6751	1.0000
61	HoVer claim verification ★	accuracy	4,000	0.8885	0.7770	1.0000
62	When2Call MCQ ★	accuracy	3,652	0.7998	0.7331	1.0000
64	New Yorker caption matching ★	accuracy	528	0.6439	0.5549	1.0000

The model repository does not include prediction/result files. These belong in a separate results Dataset under runs/lemondecider-v3-chunked-public03/; attach its published link and immutable model revision when submitting. No Dataset URL is invented here. Publication code has naming-only differences from the historical service; its planning/prediction implementations are unchanged (release notes). The current Dataset publication copy normalized response-model labels; preserve original untouched predictions for review or obtain reviewer acceptance of that transformation. A naming-cleanup rescore is not a new inference evaluation.

Usage

Install a hardware-compatible Torch 2.9.0 build in a separate Python 3.12 environment, then python -m pip install -r requirements.txt. The requirements file is a candidate recipe, not a validated NVIDIA lockfile. Clean NVIDIA compatibility has not been verified by this packaging operation. Download the whole repository at an immutable published v3 revision; custom code executes during loading, so inspect it first.

Python: native joint decisions with capacity-aware chunking

import json
import sys
from pathlib import Path

checkpoint = Path("./LemonDecider").resolve()
sys.path.insert(0, str(checkpoint / "code"))
from serve_312k import LemonDeciderHFService

service = LemonDeciderHFService(checkpoint, "cuda:0", None)
request = {
    "state": "The customer wants to return a damaged package.",
    "questions": {
        "action": {
            "type": "choice",
            "instructions": "Choose the next action.",
            "criteria": {"refund": "Offer a refund", "tracking": "Check tracking"}
        },
        "damaged": {"type": "noul", "instructions": "Is the package damaged?"}
    }
}
print(json.dumps(service.predict(request), indent=2))

CLI and HTTP

From this repository root:

python code/verify_files.py
python tests/test_chunking.py
CUDA_VISIBLE_DEVICES=0 python code/infer.py --input examples/request.json
# Alternatively, start the local HTTP service:
CUDA_VISIBLE_DEVICES=0 bash code/serve.sh
# In another terminal:
python code/smoke_test.py

HTTP endpoints: GET /health, GET /v1/models, POST /v1/systemone. The service binds to localhost by default. Use LEMONDECIDER_API_KEY for optional bearer authentication; the provided local smoke client assumes it is unset. code/serve.sh accepts PORT, DEVICE, CHECKPOINT, PYTHON and AIS_SERVER_CPU_THREADS. Never expose an unauthenticated service publicly.

A long question list is recursively bisected in its original order. Every chunk retains the complete state and options. Planning finishes before any forward; if even one singleton cannot fit, the whole request returns HTTP 422. Returned answers keep their original IDs, and input usage sums all chunks including repeated state. No reread, option pruning or benchmark-specific rendering is used. The root model.py is retained for file-layout compatibility but uses the legacy independent-question path; it is NOT the reproducing CLI for this joint v3 run.

Intended use & limitations

• Designed for option-based routing, classification, structured choices and typed decision endpoints, not open-ended text generation.
• No correctness or calibration guarantee follows from a probability output.
• Capacity is 8192 tokens per chunk, NOT the backbone's 262144-position setting.
• A long shared state cannot be shortened to make a question fit.
• Text-only service; vision tensors in the checkpoint do not imply a supported image API or a reported vision benchmark result.
• Native chunking is part of this release's inference contract, not a benchmark optimization that can be silently omitted.
• Unsupported requests remain in scoring coverage. Public scores do not establish performance on private tests or real-world deployment traffic.

License

LICENSE and LICENSE.code are preserved from the v3 export (Apache-2.0 and MIT texts respectively). Retain all applicable upstream and author notices; this packaging does not expand rights over training or evaluation data.

Acknowledgements

The Lemon project builds on the Qwen backbone and upstream typed-decision runtime work. Attribution and notices remain in the original source and license files.