GroupAssistBench

One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough

Zhanghao Hu*, Linhai Zhang*, Qian Zhao, Qi Zhu, Yuan Hua, Di Liang, Jiasheng Si, Xin Zhao, Yulan He, Zhumin Chen†, Lin Gui†

* Equal contribution. † Corresponding authors.

📄 Paper  |  💻 Code  |  🌐 Project Page

A Joint Research Project by Tencent Yuanbao Team and KCL NLP Group

Tencent Yuanbao Team  ×  KCL NLP Group, King's College London

---

Overview

GroupAssistBench is a benchmark for evaluating how LLM agents recall and use memories to assist multiple participants in real-world group conversations.

Built from 50 real-world group chats containing 80,299 messages, it includes 2,395 evaluation instances across two task levels:

• L1 — Memory Recall: Recovering historical information with reference answers.
• L2 — Memory Use: Providing open-ended assistance grounded in group memories, evaluated using instance-specific rubrics informed by real user feedback.

Both levels cover three memory dependencies: single (M1), multiple (M2), and conflicting (M3) memories.

Dataset Statistics

	M1	M2	M3	Total
L1	776	448	150	1,374
L2	221	650	150	1,021
Total	997	1,098	300	2,395

Dataset Files

The dataset/ directory contains:

• items.jsonl: Evaluation queries, L1 reference answers, and L2 rubrics.
• items_full.jsonl: Full construction records and supporting memory anchors.
• contexts.json: Anonymised Chinese group conversation histories.
• contexts_en.json: English reference translation of the group conversations (translated using GPT-5.5).
• cutoffs.json: Per-instance history cutoffs to prevent future information leakage.
• profiles.json: Group member profiles.
• manifest.json: Instance IDs grouped by task and memory dependency.
• dataset_meta.json: Dataset statistics and checksums.

Note: The data files use D1/D2/D3, corresponding to M1/M2/M3 in the paper.

English Reference Translation. We additionally provide contexts_en.json, an English translation of the original Chinese group conversations using GPT-5.5, to improve accessibility for international researchers. Both versions share the same group keys, message order, and turn indices. The original Chinese conversations (contexts.json) and evaluation instances (items.jsonl) remain the primary benchmark data, while the English translation is provided for reference only.

Usage

load_dataset loads the 2,395 evaluation instances (dataset/items.jsonl). The conversations the instances are asked against are in dataset/contexts.json. That file is a dictionary keyed by group rather than a table, so download it with huggingface_hub:

import json
from datasets import load_dataset
from huggingface_hub import hf_hub_download

items = load_dataset("tencent/GroupAssistBench", split="test")
print(len(items))  # 2395

path = hf_hub_download("tencent/GroupAssistBench", "dataset/contexts.json", repo_type="dataset")
with open(path, encoding="utf-8") as f:
    contexts = json.load(f)  # {group: [[turn, speaker, text], ...]}

item = items[0]
history = [m for m in contexts[item["group"]] if m[0] < item["cut_idx"]]
# history: the conversation a system may read before the request
# item["query"]: the request
# item["gt"] / item["gt_atoms"] (L1) or item["criteria"] (L2): the reference for scoring

To download all files, including contexts_en.json (English reference translation), profiles.json and items_full.jsonl:

from huggingface_hub import snapshot_download

local_dir = snapshot_download("tencent/GroupAssistBench", repo_type="dataset")

or from the command line:

hf download tencent/GroupAssistBench --repo-type dataset --local-dir GroupAssistBench

For the complete dataset construction pipeline, evaluation protocol, and implementation details, please refer to our GitHub repository.

License

The dataset is released under CC BY-NC 4.0 for non-commercial use.

See LICENSE_Dataset.txt for details.

Citation

@misc{hu2026assistantmemoriesbenchmarkingrealworld,
  title={One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough},
  author={Zhanghao Hu and Linhai Zhang and Qian Zhao and Qi Zhu and Yuan Hua and Di Liang and Jiasheng Si and Xin Zhao and Yulan He and Zhumin Chen and Lin Gui},
  year={2026},
  eprint={2611.04867},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2611.04867}
}