Spaces:
Configuration error
Configuration error
| license: mit | |
| # Precise Debugging Benchmarking (PDB) | |
| ⭐ **Neurips'26: Evaluations & Datasets** | |
| 📄 [Paper](https://arxiv.org/abs/2604.17338) · | |
| 💻 [Code](https://github.com/Bill1235813/PDB) · | |
| 🌐 [Project page](https://precise-debugging-benchmark.github.io/) · | |
| 🏆 [Leaderboard](https://precise-debugging-benchmark.github.io/leaderboard.html) | |
| **PDB** is an automatic pipeline that turns any coding dataset into a | |
| *debugging* benchmark with fine-grained metrics. Beyond binary unit-test | |
| scores, PDB evaluates a debugger with **edit-level precision** (did the model | |
| touch only the lines it had to?) and **bug-level recall** (did it fix every | |
| fault?). This rewards targeted fixes and penalizes the regeneration behavior | |
| frontier LLMs often fall back on. | |
| > Frontier models like GPT-5.1-Codex and DeepSeek-V3.2-Thinking top unit-test | |
| > leaderboards (>76%) but score at or below 45% on precision: they pass tests | |
| > by rewriting, not repairing. PDB makes that gap measurable. | |
| ## Released datasets | |
| | Dataset | Size | Bug granularity | Notes | | |
| |---|---|---|---| | |
| | [PDB-Single](https://huggingface.co/datasets/Precise-Debugging-Benchmarking/PDB-Single) | 5,751 | single line | main evaluation set: tasks not easily solved by 7+ of 9 reference models | | |
| | [PDB-Single-Full](https://huggingface.co/datasets/Precise-Debugging-Benchmarking/PDB-Single-Full) | 7,589 | single line | full initial pool before easy-case filtering | | |
| | [PDB-Wild](https://huggingface.co/datasets/Precise-Debugging-Benchmarking/PDB-Wild) | 484 | multi-line / repository-level | PDB-Multi (256) + SWE-smith repository bugs (228) | | |
| | [PDB-Multi](https://huggingface.co/datasets/Precise-Debugging-Benchmarking/PDB-Multi) | 256 | 2–4 line blocks | BigCodeBench/LiveCodeBench part of PDB-Wild; programs with ≥35 LOC | | |
| | [PDB-Results](https://huggingface.co/datasets/Precise-Debugging-Benchmarking/PDB-Results) | – | – | model outputs and scores | | |
| PDB-Single, PDB-Single-Full and PDB-Multi are derived from [BigCodeBench](https://huggingface.co/datasets/bigcode/bigcodebench) | |
| and [LiveCodeBench](https://huggingface.co/datasets/livecodebench/execution); PDB-Wild adds | |
| [SWE-smith](https://github.com/SWE-bench/SWE-smith) repositories. All are built by the PDB | |
| pipeline and evaluated with precision / recall / unit-test pass rate. | |
| ## Citation | |
| ``` | |
| @article{chai2026pdb, | |
| title={Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?}, | |
| author={Chai, Miaosen and Zhu, Wang Bill and Wang, Shangshang and Liu, Yejia and Bian, Song and Dong, Honghua and Neiswanger, Willie and Jia, Robin}, | |
| journal={arXiv preprint arXiv:2604.17338}, | |
| year={2026} | |
| } | |
| ``` | |
| ## Contact | |
| Questions / submissions: wangzhu@usc.edu, miaosenc@usc.edu. | |