utahnlp/knows-benchmark

KNOWS Benchmark

KNOWS evaluates web agents on the work people actually do in Google Workspace: writing documents, building spreadsheets, and composing slide decks that require web research, multi-step tool use, and faithful grounding in retrieved sources.

This dataset contains the task definitions — the prompt an agent receives, plus the structured evaluation rubric used to grade the artifact it produces.

Tasks	110 (22 templates × 5 instances)
Domains	20
Mean prompt length	204 words
Evaluation steps per task	~25

Artifact	Templates	Tasks
Google Docs	5	25
Google Sheets	9	45
Google Slides	8	40

• Code and evaluators: https://github.com/alexgill321/KNOWS-benchmark
• Project page: https://alexgill321.github.io/KNOWS-benchmark

This dataset does not score anything

Each task is graded by a task-specific Python evaluator that inspects the finished artifact through the Google Workspace APIs and its rendered form — combining deterministic comparisons, tolerance-based matching, geometric layout tests, document-structure inspection, browsing-trace verification, and LLM/VLM judgements. Those evaluators live in the GitHub repository, not here.

To actually run and score the benchmark, follow RUNNING_STANDALONE.md, or use the reference harnesses: BrowserGym-Knows and AgentLab-Knows.

Use this dataset when you want the prompts and rubric as data — for analysis, task selection, or driving your own harness.

Fields

field	description
task_id	Registered harness id, e.g. knows.docs_1_formal_letter.1. For nine families this differs from family (sheets_6_investmenttracker registers as knows.sheets_6_stock_tracker) — use task_id to launch a task, family to find it on disk
short_id	Short internal id (e.g. 11a); null where none was assigned
family	Task template, e.g. docs_11_personal_recipe_ocr
instance, instance_num	Which of the five instances
artifact	Docs, Sheets, or Slides
group, domain	Use-case grouping and domain label
prompt	The agent-facing instruction, verbatim from task.md (which itself carries {{...}} placeholders where a task writes to Drive — see below)
checkpoints	Rubric: [{n, title, description, steps[]}], each step carrying name, description, text, category, llm_vlm_primary
n_steps	Number of evaluation steps
total_points	Points available for the task
requires_browsing_history	Whether the evaluator checks the agent's visited URLs
source_assets	Hosted, view-only Drive files the prompt references — usable as-is
evaluator_url, checkpoints_url	Links to the evaluator and rubric on GitHub

Step category labels come from the benchmark's step-type taxonomy (structural_check, information_retrieval, formatting_check, content_check, spatial_check, web_visit_check, visual_check), with llm_vlm_primary marking steps decided primarily by a model judge.

External assets

Six task families reference Google Drive files from their prompts, in two kinds.

Files the task reads. Listed in source_assets (30 across the benchmark) and left in the prompt verbatim. These are hosted copies shared anyone-with-link, view only, so they work as-is and a benchmark run cannot modify them.

Locations the task writes into. Ten tasks — all of sheets_10_paper_sorting and slides_17_removeimagesaddplaceholders — ask the agent to upload files into a Drive folder, or to edit a scratch copy of a deck. A write destination cannot be shared between users, so the benchmark does not ship one: the prompt carries a {{TOKEN}} you replace with a location in your own Drive.

placeholder	what to supply
{{OUTPUT_FOLDER_URL}}	A folder the agent may write into. For sheets_10 it must contain pdfs/ and figures/ subfolders — the prompt names those by name
{{IMAGES_FOLDER_URL}}	A folder for the images extracted from the deck
{{WORKING_COPY_URL}}	An editable copy of the source presentation (slides_17 instance 1 only)

The code repository provisions all of them for you, for both families at once:

python src/browsergym/knows/eval/tasks/provision_run_targets.py --all

It creates a fresh folder tree per run and writes the URLs to run_targets.json, which the harness and the evaluators both read — so the folder the agent uploaded to is the one that gets graded. On your own harness, substitute with a plain string replace on prompt. Tasks whose prompt contains no {{ need no setup at all. See ASSETS.md for the full layout.

Maintenance and versioning

Tasks are curated to be time-agnostic and independent of any single website, but some reference live pages that can change. External sources the evaluators depend on are verified periodically; if a change significantly alters a task's difficulty or solvability, the task is updated or replaced with one of similar scope. Every change ships as a new tagged release on GitHub, and prior versions stay available, so results remain reproducible. Please report the benchmark version alongside your results. Issues can be reported via the templates.

This dataset corresponds to benchmark v1.1.0.

Licence

Apache-2.0, matching the code repository.

Citation

@inproceedings{gill2026knows,
  title     = {The Hard Part Comes After Search: Benchmarking Web Agents on
               Synthesizing, Organizing, and Displaying Knowledge},
  author    = {Gill, Alexander and Ishmam, Md Farhan and Nguyen, Xuyen and
               Bhat, Neha and DeYoung, Parker Henry and
               Hashemi Chaleshtori, Fateme and Stringham, Nathan and
               Marino, Kenneth and Marasovi\'{c}, Ana},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}

Please also state which benchmark version you ran (above).