SQL2Text: a semantic captioning benchmark

Test sets for SQL2Text, the task of describing a SQL query in natural language (semantic captioning, the reverse of Text2SQL). They were introduced in

Ali Al-Lawati, Jason Lucas, Prasenjit Mitra. Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text. COLING 2025.

A SQL query can be described correctly in many ways, so a single reference question, as found in Text2SQL datasets, is a weak target for the reverse task. Each query here therefore comes with several reference captions: three written by GPT-4o through an iterative generate, review and refine prompt, plus the original human-written question from the source dataset where that question passed validation.

Contents

Three configurations, each with a single test split.

Configuration	Source dataset	Queries	Generated captions	Original questions	Total references
cosql_s2t	CoSQL, dev	290	870	280	1,150
sparc_s2t	SParC, dev	289	867	282	1,149
spider_s2t	Spider, dev	274	822	273	1,095
Total		853	2,559	835	3,394

from datasets import load_dataset

ds = load_dataset("aliwister/SQL2Text", "spider_s2t", split="test")
print(ds[0])
# {'idx': 'spider-0',
#  'query': 'SELECT count(*) FROM singer',
#  'utterances': ['How many singers are there?',
#                 'Please provide the total number of singers.',
#                 'Can you count the total number of singers?',
#                 'How many singers do we have?'],
#  'query_type': 'aggregate'}

Fields

Field	Type	Description
idx	string	Identifier of the example, for example cosql-17. Identifiers are unique within a configuration and are not contiguous, because some examples were removed during validation.
query	string	The SQL query to be captioned.
utterances	list of strings	The reference captions. The first three were generated by GPT-4o. When there is a fourth, it is the original question from the source dataset; it is absent where the original question did not pass validation.
query_type	string	simple, aggregate or nested, the grouping used for the per-type results in the paper.

How the data was built

1. Source. Query and question pairs from the development sets of the three source datasets: all 293 examples of CoSQL and random samples of 293 examples each from SParC and Spider. For the conversational datasets CoSQL and SParC, the pair is the final query and utterance of each interaction.
2. Generation. For every query, GPT-4o followed a three-step prompt: generate three captions, review each one for accuracy, clarity, fluency and relevance to the query, then refine them using that feedback. The three references kept per query were chosen by AlignScore and BERTScore against the original question.
3. Validation. Three evaluators with NLP and database experience checked every query together with its original question and generated captions for semantic accuracy, preservation of the SQL logic and technical precision (Fleiss' kappa 0.802). Inconsistent queries, original questions and captions were removed, which left the 853 queries above.

The prompt and the full protocol are in the paper.

Use

The paper evaluates captions produced by a model for each query against the references in utterances, with BLEU-4, BERTScore and AlignScore. Few-shot demonstrations in the paper are drawn from the training sets of the source datasets, which are not redistributed here. Code for the experiments, including the graph-aware demonstration selection (AST-ICL), is at github.com/aliwister/ast-icl.

Source datasets and licence

The queries and original questions come from Spider, SParC and CoSQL, all distributed under CC BY-SA 4.0. This dataset is released under the same licence. Please also cite the source datasets when using it.

Citation

@inproceedings{al-lawati-etal-2025-semantic,
    title = "Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for {SQL}2{T}ext",
    author = "Al Lawati, Ali  and
      Lucas, Jason  and
      Mitra, Prasenjit",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.coling-main.536/",
    pages = "8026--8042"
}