XinNUS/DOVE-Bench

DOVE: Distinctive Occurrence-level Video-caption Evaluation

DOVE asks whether a video model can describe one specified occurrence of a repeated event precisely enough to tell it apart from the other occurrences in the same video. The model receives the complete source video, one target time interval, and a fixed prompt. The sibling occurrences and every answer key stay hidden while it writes. A text-only judge then checks the caption against frozen human-verified probes.

Introduced in Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding.

• 180 source-video/query groups, 530 target intervals, 180 full source videos
• 1,223 human-verified temporal probes: 469 Order, 420 State, 334 Outcome
• input protocol full_video_single_target_interval_v1
• version v1 (dove-v1-180g-530q), frozen; later releases get a new version id

Why it is hard

Within a group, the same query occurs in two to five non-overlapping intervals of one video. The siblings are hard distractors by construction: they share the query and differ only in fine-grained detail. A caption that merely restates the query matches every occurrence and therefore scores zero. The caption has to state what makes this occurrence identifiable — the order of actions, how a person or object changed, or the visible outcome that marks it.

Layout

data/public/questions.jsonl          530 model-visible inputs
data/public/*_videos.txt             source video lists per upstream dataset
data/private/temporal_probes.jsonl   1,223 scoring probes with answer keys
videos/<dataset>/*.mp4               180 full source videos
scripts/run_dove.py                  inference / judge / score entry point
scripts/compare_models.py            paired group-level bootstrap
adapters/example_adapter.py          adapter template
VERSION.json, CHECKSUMS.sha256       frozen snapshot metadata

data/private/ is published, but "private" is literal about the evaluated model: never put the probes in the model's context. They carry the answers. Keeping them out of generation is what the benchmark measures.

Quick start

hf download XinNUS/DOVE-Bench --repo-type dataset --local-dir DOVE-Bench
cd DOVE-Bench && pip install -r requirements.txt
python3 scripts/run_dove.py validate

Evaluate predictions from any framework

Write one JSON object per line, covering all 530 public qids exactly once:

{"qid":"tcb_0_0","caption":"The person pours water into the white bowl, then places its lid on."}

python3 scripts/run_dove.py validate --predictions results/my_model/predictions.jsonl
python3 scripts/run_dove.py judge \
  --predictions results/my_model/predictions.jsonl \
  --output results/my_model/judgments.jsonl \
  --base-url http://127.0.0.1:18901/v1 --judge-model qwen-judge --workers 32
python3 scripts/run_dove.py score \
  --judgments results/my_model/judgments.jsonl \
  --output results/my_model/report.json

The judge stage is append-only and resumable; failed rows stay retryable.

Or run a model through an adapter

Copy adapters/example_adapter.py and implement create_predictor(model_args). The adapter receives video and prompt only, and must not read the probes.

python3 scripts/run_dove.py infer --adapter adapters/my_model.py \
  --model-name my-model --output results/my_model/predictions.jsonl

Shard with --num-shards N --shard-id I, then merge by qid before validating.

The judge is part of the protocol

Reported numbers are only comparable when the judge is identical. The published results use a text-only Qwen/Qwen2.5-72B-Instruct served through an OpenAI-compatible endpoint, with temperature=0, max_tokens=96 and a 16,384-token context, under the system prompt in scripts/run_dove.py. The judge never sees the video: it decides only whether the caption supports each probe's correct choice. For example:

vllm serve Qwen/Qwen2.5-72B-Instruct --served-model-name qwen-judge \
  --tensor-parallel-size 2 --max-model-len 16384 --port 18901

Changing the judge, its decoding settings, the prompt, or the probe snapshot makes a new protocol version. Do not mix those numbers with the table below.

Metrics

Metric	Meaning
TEC	macro-average of Order, State and Outcome coverage (headline)
Temp-Cov	micro-average over all probes
Clip-Complete	fraction of intervals whose every probe is supported
N/M	fraction of probes the caption does not mention
contradiction_rate	fraction for which the caption supports a wrong choice

529 of the 530 intervals are scored; tcb_147_1 has no retained probe.

For paired confidence intervals, resample the 180 groups as whole units:

python3 scripts/compare_models.py --probes data/private/temporal_probes.jsonl \
  --model reference=results/reference/judgments.jsonl \
  --model candidate=results/candidate/judgments.jsonl \
  --reference reference --iterations 20000 --seed 20260903 \
  --output results/reference_vs_candidate.bootstrap.json

Reference results

Table 5 of the paper, in percent. See the paper for the full discussion.

Model	Size	Temp-Cov↑	Clip-Complete↑	Order↑	State↑	Outcome↑	N/M↓	TEC↑
GPT-5.6-sol	–	17.3	5.9	16.4	17.9	17.7	71.1	17.3
TRACE-Uni	–	2.0	0.6	1.3	1.9	3.0	95.7	2.1
Sa2VA	8B	0.0	0.0	0.0	0.0	0.0	99.9	0.0
VideoChat2-HD	7B	1.8	0.4	1.7	1.7	2.1	93.5	1.8
OMTG	4B	0.7	0.2	0.6	1.0	0.6	98.8	0.7
VideoChat-R1	7B	3.2	0.4	2.8	2.9	4.2	93.9	3.3
Time-R1	7B	2.6	0.6	2.3	2.9	2.7	94.8	2.6
TimeLens	7B	4.4	0.9	3.8	4.3	5.4	90.8	4.5
LLaVA-Video	7B	8.3	2.5	8.3	8.3	8.1	81.1	8.2
Video-LLaMA3	7B	1.0	0.6	0.4	1.2	1.5	97.5	1.0
InternVL3	8B	2.4	0.6	3.0	1.4	2.7	94.1	2.4
Open-o3-Video	4B	13.5	4.7	11.5	14.8	14.7	76.2	13.6
Qwen3-VL	4B	14.7	5.5	12.6	17.1	14.7	75.1	14.8
+ Ours	4B	33.4	14.6	31.8	37.4	30.8	50.7	33.3

Even the strongest general model mentions nothing relevant for about seven of ten probes, which is the gap DOVE is built to expose.

How it was built

Groups were sampled in equal proportion from Cosmos-Cap, Moment-10M and VTimeLLM, excluding videos that appear in training or existing test splits, and keeping only videos where one query has two to five valid, non-overlapping, non-redundant occurrences. Gemini 3 Pro saw all occurrences of a group at once and proposed, per occurrence, whether the query occurs, which facts are shared versus specific, and plausible-but-false statements. Those proposals were treated as suggestions only: annotators verified every event, fact, label and negative, then re-checked each group as a whole to fix cross-occurrence inconsistencies and drop anything vague or unsupported. The Order/State/Outcome probes were then written to require multi-frame evidence and a unique answer, and frozen before any evaluation.

Provenance and licenses

• Source videos come from Cosmos-Cap, Moment-10M and VTimeLLM and remain under their original terms; data/public/*_videos.txt lists which ones are used.
• Annotations, probes and evaluation code in this repository are released under CC BY 4.0.
• Identifiers and schema_version strings keep the historical tcb_ prefix so that results stay comparable with earlier runs. DOVE is the public name.

Citation

@article{generalizedcyclegrpo,
  title   = {Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding},
  author  = {Zhang, Xin and Wang, Haochen and Lin, Beibei and Zhang, Zhaoxiang and Tan, Robby T. and Li, Jason and Tao, Dacheng},
  journal = {arXiv preprint arXiv:2610.29552},
  year    = {2026}
}