acecode87k / README.md
guanning's picture
AceCode-87K train / held-out 1k test split used for the Qwen3-1.7B-Base code RL runs
de32d94 verified
|
Raw History Blame Contribute Delete
2.25 kB
metadata
license: mit
task_categories:
  - text-generation
tags:
  - code
  - reinforcement-learning
  - rlvr
configs:
  - config_name: default
    data_files:
      - split: train
        path: train.parquet
      - split: test
        path: eval.parquet

AceCode-87K train / held-out test split (verl format)

The exact train and test data used for the binary-reward code RL runs on Qwen3-1.7B-Base (GRPO / MaxRL) in max-rl.

Source: TIGER-Lab/AceCode-87K (AceCoder, arXiv:2502.01718), which ships a single train split: function-level Python problems, each with GPT-4o-mini-synthesized assert test cases.

file split rows data_source
train.parquet train 85,847 acecode
eval.parquet test (held-out) 1,000 acecode_eval

Train and test share no AceCode id (overlap 0). Per-split counts are in stats.json.

How the split was made

preprocess_acecode_data.py (included) on the full 87,149 rows:

  1. shuffle with seed 42;
  2. keep only assert ... test strings; drop rows with < 3 tests, empty questions, or a duplicate question (302 rows dropped); keep at most 16 tests per prompt;
  3. drop prompts longer than 3,072 tokens under the Qwen3-1.7B-Base chat template (0 dropped);
  4. the last 1,000 surviving rows are the test split, the rest are train.

Schema (verl RL parquet)

  • prompt: chat messages, one user turn = the AceCode question + the suffix "\nPlease reason step by step, then write a self-contained Python solution. Give your final program in a single ```python code block."
  • ability: "code"
  • reward_model: {"style": "rule", "ground_truth": "{\"tests\": [\"assert ...\", ...]}"}
  • extra_info: split, index, acecode_id, num_tests

Reward

Strictly binary: take the last ```python block of the response, append all assert tests, run in a fresh python -I subprocess (10 s wall / CPU limit, 4 GiB address space). Score 1.0 iff the process exits 0, else 0.0 (no code block → 0.0). Validated 60/60 against AceCode-87K's own pass_rate ∈ {0, 1} inference labels.

Evaluation in the runs: 4 samples per test prompt (T=0.6, top-p 0.95), reporting mean@4 and best@4.