DesignTaste

Benchmarking AI-Generated Visual Design Against Award-Winning Works

403 design tasks | 2,418 generated images | 6 image models | 5 evaluation dimensions

DesignTaste compares AI-generated visual designs with award-winning works from 100 Beste Plakate 2025 and the ADC Annual Awards 2026. Each model receives the same reconstructed design brief without seeing the original work. Generated images and award-winning works are then scored independently for their design idea and visual expression.

GitHub | Evaluation protocol | Submission guide | Reference links and attribution | Data terms

What is included

Track	Tasks	Generated images	Rating records	Design scope
poster	100	600	700	Poster design
adc	303	1,818	2,121	20 disciplines, including branding, packaging, advertising, books, illustration, and spatial design
Total	403	2,418	2,821

Every task has one generated image from each of six models and seven rating records, including the award-winning reference. This release includes the original generated-image files, approximately 4.84 GB of image bytes, embedded in Parquet shards. It also includes reviewed briefs, task metadata, source attribution, raw dimension scores, and the benchmark rubrics.

The 403 original award images are hosted externally. The task metadata and reference-links table provide a reference_url for the selected source image and a work_url for its official work page. Original award-image bytes and comparison composites are not included in this repository. Source links do not grant permission to reuse the linked material.

Quick start

Stream generated examples without downloading the entire collection:

from datasets import load_dataset

generated = load_dataset(
    "ArtArk/DesignTaste", "generated", split="test", streaming=True
)
example = next(iter(generated))

image = example["image"]  # Decoded PIL image
print(example["brief"])
print(example["model_name"], example["overall_score"])

Load the lightweight task and rating tables for generation or analysis:

tasks = load_dataset("ArtArk/DesignTaste", "tasks", split="test")
ratings = load_dataset("ArtArk/DesignTaste", "ratings", split="test")

poster_tasks = tasks.filter(lambda row: row["track"] == "poster")
references = ratings.filter(lambda row: row["source"] == "reference")

assert len(tasks) == 403
assert len(ratings) == 2821
assert len(references) == 403

All three configurations have a single test split. generated is the default configuration. No training or validation split is supplied.

Data structure

Configuration	Unit of observation	Main fields
generated	One model output per task	image, brief, task identifiers, source, model_name, category context, image provenance, judge, five scores, overall_score
tasks	One design task	brief, task identifiers, reference URLs and hashes, title, creators, agency, credits, category and award metadata
ratings	One scored reference or model output	rating_id, task identifiers, source, model_name, image provenance, judge, five scores, overall_score

Use task_key, such as poster:1 or adc:62199, to join configurations. task_id preserves the original string identifier, and track is either poster or adc. A rating is identified by (task_key, source). source="reference" denotes the award-winning work. Missing or inapplicable string metadata uses an empty string, and absent role credits use the string {} in credits_json. Numeric fields can be null where information is unavailable. The complete field definitions are in metadata/schema.json.

For Poster, creators retains the source creator attribution. For ADC, creators and agency identify the credited agency, while credits_json preserves the available role-specific credits from the official award page. These fields distinguish agency attribution from individual credits. metadata_provenance, metadata_source_status, and metadata_limitations describe the metadata source and its limits. ADC spans 20 discipline values and 135 more specific category values.

image_sha256 identifies the original image file. scored_image_sha256 identifies the actual judging input. ADC judging used image previews, so these hashes can differ. The release contains the original generated images, with preview dimensions and hashes retained as provenance. Preview files are not duplicated in the upload. reference_image_count describes the original ADC project's image count, even though each benchmark task evaluates one image.

The two brief hashes use different serializations of the same text. brief_sha256 hashes the UTF-8 brief directly. annotation_brief_sha256 hashes json.dumps(brief, ensure_ascii=False, sort_keys=True, separators=(",", ":")) encoded as UTF-8. All 403 brief texts and all 2,821 annotation hashes have been checked against their respective conventions.

Evaluation and scoring

The judge receives one anonymous image, its brief, and the track-specific rubric. The main judge is gpt-6.1-sol for 2,820 records. The reference for ADC task 62340 has the source judge label codex_visual_review. The released rating tables contain scores, not the judge's textual rationales.

The five reporting names are shared across tracks, while the detailed criteria reflect their different design tasks. Raw scores were assigned as integers from 0 to 10. The Parquet score columns store numeric values on that same 0-10 scale.

Dimension / score field	Poster rubric ID	ADC rubric ID	Weight
Thematic Expression / thematic_expression	brief_communication	brief_fit	2 (20%)
Creative Originality / creative_originality	conceptual_originality	concept_strength	3 (30%)
Visual Organization / visual_organization	visual_organization	category_fit	1.5 (15%)
Aesthetic Expression / aesthetic_expression	aesthetic_expression	visual_craft	2 (20%)
Formal Completion / formal_completion	formal_resolution	visible_communication	1.5 (15%)

overall_score uses a 0-100 scale and the default weights 2:3:1.5:2:1.5:

overall_score = 10 * sum(weight_i * raw_score_i) / sum(weight_i)

Multiply an individual raw dimension by 10 to display it on a 0-100 scale. The overall score is already on that scale. Models are compared with the reference for the same task, and each task contributes equally to its track's mean. Higher/tied/lower counts compare independently assigned scores rather than direct preference votes. project_group identifies related tasks. There are 99 groups in Poster and 229 in ADC.

Released results

The following overall means are reproduced from the frozen scores with the default weights. Each model contributes 403 images, comprising 100 Poster and 303 ADC outputs. These are benchmark judgments, not the original award juries' ratings.

Model / source	source identifier	Poster	ADC
Award-winning works	reference	78.275	80.343
Seedream 5.0 Pro	doubao-seedream-5-0-pro-260628	72.440	72.566
Nano Banana Pro	nano-banana-pro	75.180	77.028
GPT-Image-2.5 Flare	gpt-image-2.5-flare	74.710	78.094
GPT-Image-2	gpt-image-2	75.060	79.477
GPT-Image-2.5 Sunburst	gpt-image-2.5-sunburst	75.840	79.705
ArtArk	artark	79.635	80.736

For new evaluations, generate one image per task from its brief and applicable category context, then use the matching rubric and scoring prompt in the submission guide. Reweighting the released dimensions requires no new image generation or judging.

Scope and limitations

The English briefs were reconstructed from public award materials and reviewed for factual grounding and room for alternative design solutions. They are not verified original client specifications. The ADC cohort retains 303 of 333 initial tasks. Thirty were excluded because one or more models declined generation under their content moderation policies.

Scores reflect one primary model judge and its rubric. A single still image cannot establish the quality of an entire campaign, design system, physical product, motion sequence, or audience experience. Award collections and the retained tasks also represent a limited range of design practice. Use the source work pages and metadata to understand each task's context, and interpret score differences within these limits.

Rights and attribution

The dataset is accompanied by the DesignTaste Data Terms and Rights Notice. The generated images are supplied for benchmark inspection and reproducibility. Applicable provider terms and rights in depicted third-party material remain relevant. Original award images remain on their external hosts, under their respective rights and source terms.

The benchmark code is MIT-licensed. That code license does not extend a blanket MIT or Creative Commons grant to this dataset's images, award metadata, or other third-party materials. Attribution is provided through the task metadata and the citation below. Report corrections through the project issue tracker.

Citation

@misc{chen2026designtaste,
  title = {DesignTaste: Benchmarking AI-Generated Visual Design Against Award-Winning Works},
  author = {Chen, Zhou and Li, JiaXiang and Pan, DanYing and Li, Ye
            and Yang, TianRui and Zhang, EnBao and Xiao, YaNan and Sun, LiXin
            and Liu, Anqi and Zhao, Dong and Li, JianYuan and Ding, YuanYuan
            and Kong, Kai and Xu, Xinwen and Du, HuiTang and Zhou, Min and Li, Zhiqiang},
  year = {2026},
  url = {https://github.com/ArtArk-AI/DesignTaste},
  note = {Benchmark code and data, version 0.1.0}
}