jev-lite

jev-lite is a typed decision model. It takes a state (free text, a JSON object, or a list of texts) and one or more typed questions:

Type	Question	Output
bool	a statement about the state	P(true) + confidence
choice	a prompt + 1–32 options	distribution over the options + argmax + confidence
score	a prompt + 1–32 ordered levels	distribution over the levels + argmax + expected level + confidence

It does not generate text. The answer is always one of the caller's own candidates, so the output always fits the schema. The model is meant to be a fast, cheap replacement for an LLM-as-a-judge on checks such as faithfulness of RAG answers, answerability, pairwise preference and rubric scoring. Not affiliated with TypeSafe AI.

Code: github.com/Nefelim584/JEV_Judge

Versions

Releases are git tags of this repository, the same as on GitHub. main is the latest.

Revision	Model	Release notes
v0.2.0 (latest)	Stage B: Stage A + public judge data; calibration included	v0.2.0
v0.1.0	Stage A: 25 public datasets, 512 tokens, no calibration	v0.1.0

This repository was called jev-lite-stage-a up to v0.1.0; the old name redirects here.

Usage

The heads are custom, so the checkpoint is loaded with the jevlite package from the code repository.

pip install "jevlite @ git+https://github.com/Nefelim584/JEV_Judge.git@v0.2.0"

from huggingface_hub import snapshot_download

from jevlite.calibration import LAYERS
from jevlite.inference import Predictor
from jevlite.schema import Request

path = snapshot_download("Nefelim584/jev-lite", revision="v0.2.0")
predictor = Predictor.from_final(path, device="cuda")            # calibrated probabilities
# predictor = Predictor.from_final(path, layers=LAYERS)          # + judge layer, for faithfulness claims only

request = Request(
    state={"documents": ["The Eiffel Tower was completed in 1889 for the World's Fair in Paris."],
           "question": "When was the Eiffel Tower finished?",
           "answer": "It was finished in 1899."},
    questions=[
        {"id": "faithful", "type": "bool", "prompt": "The answer is supported by the documents."},
        {"id": "topic", "type": "choice", "prompt": "What is the text about?",
         "options": ["architecture", "sports", "finance"]},
        {"id": "clarity", "type": "score", "prompt": "How clear is the answer?",
         "levels": ["0", "1", "2", "3", "4"]},
    ],
)
response = predictor.predict(request)   # all questions in one forward pass
print(response.answers["faithful"].probability)

Files

File	What
encoder/	the fine-tuned ModernBERT-large encoder and tokenizer (transformers format)
heads.pt	trunk and the Choice / Score / Bool heads
jevlite.yaml	model and encoding settings (encoding.max_len 8192: the setting of the reported numbers)
calibration.json	post-hoc calibration, read by Predictor.from_final (new in v0.2.0)

Model details

• Base: answerdotai/ModernBERT-large (395M), full fine-tuning.
• Architecture: packed encoding. Each question is one sequence (state + question + candidates), with one [MASK] marker per candidate. The marker hidden states go through a shared 1024 → 256 trunk and three heads.
• Length: trained at 512 (Stage A) and 1024 tokens (Stage B); runs at 8192 by default. Longer states are cut head + tail, and the response sets meta.truncated.
• Calibration: a temperature per primitive (default), and an optional judge layer (class-balanced Platt scaling of the Bool logit) that moves the decision threshold for faithfulness claims.
• Confidence: 1 − H(p) / log K on the calibrated probabilities. No learned confidence head yet.
• Language: English only.

Training data

Stage A (v0.1.0, replayed in v0.2.0): 25 public English datasets: FEVER, VitaminC, HoVer, MultiNLI, SNLI, WANLI, BoolQ, TabFact, QNLI, PAWS, SQuAD 2.0, Natural Questions, ClapNQ, ARC, OpenBookQA, CommonsenseQA, Cosmos QA, Social IQa, CLINC150, MASSIVE, DBpedia-14, GoEmotions, summarize_from_feedback, HelpSteer2, HelpSteer3.

Stage B (v0.2.0): judge records built by code from public data:

Dataset	Built into	License	Labels
RAGBench train: hotpotqa, finqa, tatqa, hagrid, expertqa, techqa, pubmedqa	LLM answers vs documents, whole answer and per sentence	CC BY 4.0 (sources: CC BY-SA 4.0, MIT, CC BY 4.0, Apache-2.0)	GPT-4
HotpotQA	two-paragraph claims, answerability	CC BY-SA 4.0	human + program
TAT-QA, FinQA	claims with computed numbers	CC BY 4.0; MIT / CC BY 4.0	human + program
ClapNQ	answerability (rebuilt), relevance, conciseness and completeness pairs	Apache-2.0 (NQ text CC BY-SA 3.0)	human + program
Faithfulness-QA	answers vs counterfactual passages	MIT (card)	rule-based
MiniCheck D2C	claim vs news document	MIT	GPT-4
Trust-Data (QAMPARI, ASQA rows)	preferred vs dispreferred answer, answerability	Apache-2.0	GPT-made pairs
PsiloQA (English)	LLM answer vs passage	CC BY 4.0	GPT-4o spans
Qasper	answerability on research papers	CC BY 4.0	human
RAGTruth (CNN/DM summaries)	summary vs article	MIT	human spans

No dataset under a non-commercial or research-only license was trained on. Full list with licenses and the known problems of the data: docs/DATASETS.md.

License note. The weights are released under Apache-2.0. Some training datasets are licensed under CC BY-SA, and some labels or texts were produced by LLMs (WANLI, RAGBench, MiniCheck, PsiloQA, Trust-Data). The model outputs only probabilities over caller-supplied candidates and does not generate text. All datasets are attributed above.

Training procedure (v0.2.0)

Setting	Value
Start	the v0.1.0 weights (encoder and heads)
Data	148k records per epoch × 2 (Bool 105k / Choice 22.5k / Score 20.8k), 26% replay of Stage A, max 1024 tokens
Hardware	2× T4 (Kaggle), DDP, fp16 AMP, SDPA, gradient checkpointing
Optimizer	8-bit AdamW, lr 3e-5 (encoder) / 1e-3 (trunk and heads), warmup 6%, linear decay, weight decay 0.01, grad clip 1.0
Batch	8 per GPU, effective 32, length-grouped
Steps	9,268 (2 epochs), 7 h 27 min
Loss	CE (Choice), CE + 0.5·EMD² (Score), BCE + 0.1·Brier (Bool)
Seed	42

Evaluation

All at 8192 tokens, one seed. Benchmarks were never trained on, and records whose passage occurs in the training data are removed. Baselines are zero-shot: an NLI cross-encoder (DeBERTa-v3-large) and Laya. Method: docs/EVALUATION.md.

Benchmark · metric	v0.2.0	v0.1.0	NLI	Laya
ContextualJudgeBench · consistent accuracy	0.454	0.332	0.090	0.024
HaluBench · balanced accuracy / AUROC	0.795 / 0.889	0.732 / 0.815	0.734 / 0.836	0.645 / 0.719
FaithBench · balanced accuracy (judge layer) / AUROC	0.629 / 0.681	0.579 / 0.619	0.514 / 0.481	0.480 / 0.506
RAGBench, 5 held-out subsets · balanced accuracy (judge layer) / AUROC	0.610 / 0.643	0.558 / 0.583	0.519 / 0.578	0.571 / 0.581

• ContextualJudgeBench: refusals 0.28 → 0.59, completeness QA 0.32 → 0.53. The paper reports GPT-4o at 0.512 and o1 at 0.631 (other prompts, same records).
• HaluBench: most of the gain is on HaluEval, which is built on HotpotQA, a Stage B training source. Without HaluEval the gain is +2.1 points; on the unseen CovidQA, DROP and FinanceBench it is a tie.
• Calibration (Stage A tests, not fitted on): ECE Choice 0.115 → 0.026, Bool 0.058 → 0.014, Score 0.086 → 0.036, with no answer changed.
• Typed primitives (Stage A tests, paired vs v0.1.0): Score +3.4, Bool −1.3, Choice −1.1 points.
• Speed (T4, fp16, one request per call): p50 37 ms, p95 479 ms over the held-out benchmarks; 35 ms for inputs up to 512 tokens, ~1.1 s for 6–8k tokens.

Limitations

• The decision threshold drifts per domain. Without the judge layer the model calls 93–96% of RAG answers "supported" and catches only 18–20% of the unsupported ones. The judge layer fixes most of this on RAG data but makes the model too strict on some sources (HaluBench PubMedQA −6.4, DROP −3.5 points). On a new domain, rely on the ranking or fit a threshold on a few labelled examples.
• Some of the gains are not out-of-distribution: HaluBench HaluEval shares its source with the training data, and 7 RAGBench subsets are trained on.
• Labels partly from LLMs: RAGBench (training and evaluation), MiniCheck and PsiloQA. Agreement with them is agreement with an LLM judge, not with people.
• Conciseness is weak (ContextualJudgeBench 0.20–0.29): with both answers visible the model prefers the longer one.
• Arithmetic over tables and passages is near chance outside the trained sources (HaluBench DROP, FinanceBench).
• Small regressions on the typed primitives compared with v0.1.0 (Bool −1.3, Choice −1.1, worst on reasoning multiple-choice).
• Long context: trained up to 1024 tokens; it runs and helps up to 8192, without long-context training, and long inputs are slow on older GPUs.
• Single run, single seed. Public proxies only: quality as a judge on a real production task has not been measured.

Intended use

• Fast, schema-constrained checks over text: faithfulness of an answer or claim to its context, answerability, pairwise preference, classification into a fixed label set, rubric scores.
• A cheap judge where its threshold is checked on a few in-domain labelled examples first.

Not intended for: final decisions on new domains without checking the threshold, factual-knowledge QA, arithmetic verification, or languages other than English.