Gyubeum/AndroidFlux_Evaluation_Output

AndroidFlux — Evaluation Output

Scored results for every number reported in the AndroidFlux paper, in one place.

This repository holds results only: success counts, per-episode outcomes, and the provenance needed to tell one run from another. It does not hold the benchmark inputs (source trajectories, VM snapshots, screenshots) — those live in the input repositories listed under Related repositories.

Every table in the paper can be regenerated from this repository alone:

python3 scripts/reproduce.py --check             # recount every cell and compare with its header
python3 scripts/score_tables.py --root . --all   # regenerate the paper tables

How to run them is described in scripts/README.md.

If a number printed by that script disagrees with the paper, the script is right and the paper has a typo. Known disagreements are listed in KNOWN ISSUES.

---

Layout

androidworld/<run>/                     sr_summary.json  per_task.json  MANIFEST.json
human_verified/
  replay_65/<condition>/<model>/        validation.json  per_episode.json  MANIFEST.json
                                        episodes/<task>/
  snapshot_15/<condition>/<model>/      "
llm_verified/
  replay_312/<condition>/<model>/       "
  t_plus_1/                             entry-set definition of the LLM-verified t+1 arms (no cells)
supplementary/
  t_plus_1_65/t_plus_1/<model>/         65-entry t+1 cells (not used in the tables; see KNOWN ISSUES 7)
  llm_verified_subsets/t/<model>_lv177  177- and 254-episode subset cells (see KNOWN ISSUES 6)
  replicate_runs/                       an independent replicate of one cell, for run-to-run variation
_inputs/manifests/                      the run manifests the cells were produced from
scripts/score_tables.py
scripts/scratch_80_tasks.txt
README.md

<condition> is one of:

clean   t_minus_5   t_minus_3   t_minus_1   t   t_plus_1_pos   t_plus_1_neg

Not every (axis, subset, condition, model) combination exists. score_tables.py reports missing cells rather than silently skipping them.

Per-cell files

File	Contents
validation.json	The scored cell: success_count, result_count, per-episode rows, contract-check issues, valid.
per_episode.json	One row per episode: shard, episode, source_model, task_name, is_successful, success, exception.
MANIFEST.json	Provenance: which run this is, what it was run against, what is wrong with it. See MANIFEST_SCHEMA.md.
episodes-<i>-of-<n>.parquet	Per-episode artifacts, one row per episode, payload in a files column. See Episode payload parquet. Cells whose run left outcomes only carry no such file — see KNOWN ISSUES #4.

Two notes on the layout:

• Shards are an execution detail and are flattened away. The producing harness writes shards/shard_0N/<task>/; here that is episodes/<task>/. The original shard assignment is preserved in validation.json:shard_counts and in per_episode.json:shard.
• app_data_cache/ is excluded. It is emulator scratch state, not a result.

---

Episode payload parquet

Screenshots and step records ship as parquet, not as loose files: a cell holds thousands of PNGs, and uploading them one by one runs into the hosting API's request limits.

<axis>/<subset>/<condition>/<model>/episodes-00000-of-00001.parquet

One row per episode. Identity columns (cell_id, axis, subset, condition, model_key, task_name, entry_id, source_model), the verdict as scored (is_successful, score), and the artifacts in

files  list<struct<name: string, kind: string, size_bytes: int64, sha256: string, bytes: binary>>
       kind: screenshot_png · step_json · jsonl · action_audit_png · action_audit_json
             · replay_debug_png · replay_debug_json · episode_json

name is the path the file had inside its episode directory, so the original tree can be restored exactly. sha256 is taken over the bytes AS PUBLISHED — absolute paths and machine names are scrubbed out of the text payloads first — so verification passes on the files as they stand here.

What the format guarantees

• Every file is under 5GB, the hosting limit. A cell that would exceed it is split, and the parts are named -00000-of-00003, -00001-of-00003, … The reader reads a cell as the union of its parts.
• Every row group is at most 240MB. This is what makes partial reads cheap: a reader pulling one episode loads that group, not the file. It is the row group size, not the file size, that bounds the memory a reader needs. The largest single part here is 3.3GB.
• app_data_cache/ is excluded (emulator scratch state), and so are preserved partial runs — directories whose name begins with a dot (.stopped_*, .enginedead_*_partial). Only the leading character decides, so a cell like qwen3vl_8b.dup1 is unaffected.

entry_id is the episode directory name, verbatim. Its shape follows the campaign that produced the cell, and it is not normalised — normalising it would break the correspondence with the source tree:

Subset	entry_id
snapshot_15	<source_model>__<task>__t<NNNN>
replay_65	<source_model>__<task>
t_plus_1_pos / t_plus_1_neg	<source_model>__<task>__tplus1_positive / __tplus1_negative

The key is (cell_id, entry_id); it only has to be unique within a cell. Polarity is read from the path, never from the row. In a t_plus_1 cell the stored condition is t_plus_1 for both arms, and the two arms share the same (source_model, task_name) pairs — only cell_id (and, for these campaigns, the entry_id suffix) separates them.

Reading it

python3 scripts/read_episodes.py <cell>/episodes-00000-of-00001.parquet            # what is in here
python3 scripts/read_episodes.py <cell>/episodes-00000-of-00001.parquet --extract <entry_id> --out DIR
python3 scripts/read_episodes.py <cell>/episodes-00000-of-00001.parquet --verify   # recompute every sha256

---

Condition names ↔ paper notation

The paper uses compact math notation; the repository uses filesystem-safe names.

Repository	Paper	Meaning
clean	Scratch	Task run from step 0. No replay, no primed history.
t_minus_5	$t{-}5_{\text{rep}}$	Replay through $t_{\mathrm{err}}-5$, then the policy acts.
t_minus_3	$t{-}3_{\text{rep}}$	Replay through $t_{\mathrm{err}}-3$.
t_minus_1	$t{-}1_{\text{rep}}$	Replay through $t_{\mathrm{err}}-1$.
t (in replay_65, replay_312)	$t_{\text{rep}}$	Replay through the first-error step.
t (in snapshot_15)	$t_{\text{snap}}$	VM snapshot restored to the first-error step.
t_plus_1_pos	$t{+}1^{+}$	Inject the first action of a recovery rollout that eventually succeeded.
t_plus_1_neg	$t{+}1^{-}$	Inject the first action of a recovery rollout that eventually failed.

Superscripts describe the outcome of the rollout that supplied the action, not a claim that the single action is sufficient.

Three paper columns are derived and have no directory of their own:

Paper	Definition
$t_{\text{all}}$	$(\texttt{replay_65/t} + \texttt{snapshot_15/t}) / 80$
$t{+}1_{\text{all}}$	$(\texttt{t_plus_1_pos} + \texttt{t_plus_1_neg}) / 104$
Scratch (80) in tab:aw-vs-recovery	The same AndroidWorld 116-task run restricted to the 80 recovery tasks in scripts/scratch_80_tasks.txt

The 80 recovery tasks are exactly replay_65 (65) ∪ snapshot_15 (15); the two sets are disjoint and their union is a subset of the AndroidWorld 116.

Source-name mapping, for anyone cross-referencing the older input repositories:

Repository	Original condition name
clean	clean / snapshot_clean
t_minus_1	pre_error_t_minus_1 / snapshot_pre_error_t_minus_1
t_minus_3, t_minus_5	pre_error_t_minus_3, pre_error_t_minus_5
t	error_t / snapshot_error_t
t_plus_1_pos, t_plus_1_neg	t_plus_1_positive, t_plus_1_negative

---

Scoring: two rules, and why

The two result formats in this repository are scored differently. This is deliberate, and the difference is visible only where a partial score of 0.5 exists.

Format	Field read	Success means	0.5 counts as
validation.json (human-verified, LLM-verified)	rows[].score	score > 0	success
sr_summary.json (AndroidWorld)	["total"]["successes"]	score >= 1	failure

Why they differ: the two numbers are produced by two different harnesses, and each paper table reports the figure its own harness reported. AndroidWorld's sr_summary.json counts a task correct only on full completion, which is the convention that benchmark's published numbers use. The recovery harness records partial credit and treats any progress as a recovery success, which is the quantity the recovery tables are about. Rescoring one to the other's rule would make our numbers incomparable with published AndroidWorld results.

score_tables.py applies the rule that matches the file it is reading, and every MANIFEST.json records both success (its own rule) and success_alt_rule (the other rule) so the gap is never hidden. Worked examples, all from cells in this repository:

human_verified/replay_65/t/qwen3vl_2b          scores {0.0:57, 0.5:1, 1.0:7}
                                               score>0  -> 8/65 = 12.3  <- paper
                                               score>=1 -> 7/65 = 10.8

human_verified/replay_65/t_plus_1_pos/ma3_32b  scores {0.0:29, 0.5:1, 1.0:22}
                                               score>0  -> 23/52 = 44.2  <- paper
                                               score>=1 -> 22/52 = 42.3

androidworld/ma3_7b_4env_gen1                  score points total 78.5
                                               score>=1 -> 78/116 = 67.2  <- paper
                                               score>0  -> 79/116 = 68.1

Two further traps that cost us real time:

• Read d["total"]["successes"], not the top-level key and not score_points. score_points is the sum of partial scores (78.5 above), not a count.
• Recompute from rows, never trust the header. Five LLM-verified subset cells carry a result_count and success_rate from before a denominator correction while rows is already corrected — three different numbers in one file. See KNOWN ISSUES #6. score_tables.py recomputes from rows and asserts agreement with the header, failing loudly on mismatch.

---

Denominators

Denominators are fixed constants, not derived from how many result rows happen to be present. A cell with a missing episode is a defect to be reported, not a smaller denominator.

Axis	Subset	Conditions	n
AndroidWorld	aw_116	—	116
AndroidWorld	aw_116 restricted	Scratch (80)	80
Human-verified	replay_65	clean, t_minus_5, t_minus_3, t_minus_1, t	65
Human-verified	replay_65	t_plus_1_pos, t_plus_1_neg	52
Human-verified	snapshot_15	clean, t_minus_1, t	15
LLM-verified	replay_312	clean, t	312
—	derived	$t_{\text{all}}$	80
—	derived	$t{+}1_{\text{all}}$	104

AndroidWorld is 116 including the 6 exception episodes. They are failures, not exclusions.

$t{-}3$ and $t{-}5$ also have denominator 65. When the source trajectory is too short to admit the requested offset, the task is retained and that condition is mapped to a clean start — it is not dropped. This keeps all conditions paired over the same 65 cases, at the cost of mixing true offset starts with clean starts:

Condition	Offset	Feasible	Clipped	Feasible share
t_minus_1	1	65	0	100.0%
t_minus_3	3	39	26	60.0%
t_minus_5	5	21	44	32.3%

The feasible-only figures are carried per cell in validation.json:t_minus_n and in MANIFEST.json:score.t_minus_n. The paper reports the 65-case aggregate; treat the feasible-restricted score as a secondary diagnostic, and never compare a 65-aggregate against a feasible-only figure.

---

scripts/score_tables.py

# every paper table
python scripts/score_tables.py --all

# one table
python scripts/score_tables.py --table tab:hvlvcomparison
python scripts/score_tables.py --table tab:aw-vs-recovery
python scripts/score_tables.py --table tab:instreason

# one cell, with the full derivation printed
python scripts/score_tables.py --cell human_verified/replay_65/t_plus_1_pos/ma3_32b --explain

# machine-readable
python scripts/score_tables.py --all --format json > tables.json
python scripts/score_tables.py --all --format latex

# fail the build if any cell disagrees with its MANIFEST, or any manifest is incomplete
python scripts/score_tables.py --verify

--explain prints the numerator, denominator, scoring rule, score histogram, and the MANIFEST.json fields that identify the run:

human_verified/replay_65/t_plus_1_pos/ma3_32b
  rule          score > 0            (validation.json:rows[].score)
  scores        0.0 x29   0.5 x1   1.0 x22
  success       23 / 52  = 44.23 %   -> tab:hvlvcomparison  t+1^+
  alt rule      22 / 52  = 42.31 %   (score >= 1)
  valid         false    issues: memory_prefix 14, start_state_contract 14, +4
  accepted      true
  input rev     Gyubeum/AndroidFlux_tplus1_inputs @ 0360552
  known issues  coord_mode_from_source_traj (affects score)
                memory_prefix_short_by_one  (affects score)

--verify is the gate to run before publishing: it recomputes every cell from rows, checks it against the header fields and against MANIFEST.json, checks that denominators match the table above, and checks that every cell has a manifest. It exits non-zero on any disagreement.

Identifying a cell

Do not identify a cell by model_key, and do not identify it by its score. Both fail:

model_key is not unique. Across the 321 published cells, five keys are shared by more than one model directory. In four of the five, the variant suffix is silently dropped:

model_key	directories that share it
gemma_31b_improved	gemma_31b_improved, gemma_31b_improved_think_on
gemini_3_0_flash_preview_qwen_prompt	gemini_3_0_flash_preview_qwen_prompt, ..._think_on
gemma_4_31b_it_0716_improved_m3a	gemma_4_31b_it_0716_improved_m3a, gemma_4_31b_it_improved_m3a_think_off
mobile_run_gf30	gemini_3_0_flash_preview_qwen_prompt, ..._think_off
guiowl15_8b_think	guiowl15_8b_think, yemo_guiowl15_8b_think

So gemma_31b_improved_think_on (148/312) and gemma_31b_improved (139/312) are two different cells reporting the same model_key — and thinking-enabled and thinking-disabled rows of tab:instreason are exactly the pair that key cannot distinguish.

dir_name is not reliable either. Five directories carry different model_key values in different cells, so the mapping is not even a function in the other direction:

directory	model_key values observed
gemini_3_0_flash_preview_qwen_prompt	gemini_3_0_flash_preview_qwen_prompt, mobile_run_gf30
gemini_3_0_flash_preview_qwen_prompt_think_off	gemini_3_0_flash_preview_qwen_prompt_think_off, mobile_run_gf30
gemma_4_31b_it_improved_m3a_think_off	gemma_4_31b_it_0716_improved_m3a, gemma_4_31b_it_improved_m3a_think_off
mai_ui_8b	mai_ui_8b, mai_ui_8b_history
mai_ui_2b	mai_ui_2b, mai_ui_2b_history

Matching on the score value is worse still — it crosses model boundaries. Searching for a 15/52 cell finds t_plus_1_pos/qwen3vl_2b when the target was t_plus_1_neg/qwen3vl_4b. Fix the (model, condition) pair first, then read the value; never the reverse.

The identifier is therefore the path, MANIFEST.json:identity.cell_id. dir_name and model_key are both recorded, separately, precisely because they disagree with each other.

---

KNOWN ISSUES

These are defects we have confirmed and chosen to publish rather than quietly fix. Each is recorded per cell in MANIFEST.json:status.known_issues[] with an affects_score flag, so a consumer can filter programmatically.

1. LLM-verified t: 21 cells, 20 contract issues each, on the same 5 episodes

All 21 full-set LLM-verified t cells were run before the 2026-09-07 metadata regeneration. Each reports valid: false with an identical signature:

executed_prefix 5 + memory_prefix 5 + skipped_replay 5 + history_or_budget_contract 5 = 20

That is four checks failing on the same five episodes — the "20" is an issue count, not a cell count or an episode count. The five failing episodes are the same five in all 21 cells, with zero variation in the key set:

gemini_flash_4env            | FilesMoveFile
gemini_flash_4env_gen2       | ExpenseAddMultipleFromGallery
qwen3vl_2b_4env              | Do_I_have_any_events_October_28...
qwen3vl_30b_a3b_4env         | What_incomplete_tasks...
qwen3vl_8b_think_4env_gen1   | RecipeDeleteSingleWithRecipeWithNoise

Because the failing episodes are identical across models, this is an input-side defect: row-to-row comparison within the table still holds, absolute values do not. Some cells carry additional per-model issues on top of the shared 20 (agent_rollout_error, shard_result_sizes); those are model- or run-specific and are listed per cell in MANIFEST.json:status.issues_by_type.

Do not read valid: false here as "a minor 5-episode problem." The underlying defect is larger than what validation.json catches: replay_metadata's memory.steps is empty for 103 of 312 entries (33.0%), concentrated in four source trajectories (gemma_31b_m3a_gemma31b_improved_0716_4env 49, gemma_31b_m3a_gemma_31b_m3a_4env 42, ui_tars_7b_sft gen3/gen4 6 each). The contract check catches five of them. The two numbers come from different checks and are not comparable.

The cause for the Gemma sources is confirmed: the metadata's image_name values (step_000.png) do not match the files on disk (screenshot_N.png), so every image reference resolves to nothing and memory.steps is built empty. The original trajectories are intact — this is a metadata build failure, not missing data, and it is regenerable without GPU. The ui_tars sources have a different, still-unidentified cause: their images are all present.

Checking this yourself: read len(m["memory"]["steps"]). len(m["memory"]) is the number of keys (8) and will look fine.

2. llm_verified/replay_312/t/gemma_31b_improved_think_on: 142/312 rollout errors

45.5% of this cell's episodes ended in agent_rollout_error. Its reported success rate of 47.44% is a measurement failure, not a performance figure.

This cell is marked canonical: false in its manifest. It is published because the paper's tab:instreason LLM-verified Reasoning column (47.4) rests on it, and suppressing the cell would hide that.

3. t_plus_1 32B cells: run before the replay_utils.py:747 fix

Six cells — t_plus_1_pos and t_plus_1_neg for gui_owl_32b, ma3_32b, and qwen3vl_32b — were produced on a tree that predates the replay_utils.py:747 fix (extract_action_payload was missing its AndroidWorld-native action_type branch, so t+1 candidate actions could not be extracted). All six report issues: 32:

negative arm   memory_prefix 16 + start_state_contract 16
positive arm   memory_prefix 14 + start_state_contract 14  (+ 4 unrelated)

The failing episode keys are identical across all three models within an arm (16/16 and 14/14 intersection), which places the cause on the input side, not the model. The mechanism is a memory prefix one step short of expected; 11 of the 13 start-state contract checks still pass, with only history_prefix_exact and history_last_is_live_target failing.

Root cause, recorded in the cells' own notes: the input revision (AndroidFlux_tplus1_inputs @ 03605523705a59dce1e1b3b0ba5e30881cfa5e5f) predates the candidate_coord_mode split, so coord_mode was assigned from the source trajectory rather than from the recovery action. The evidence is that the positive and negative arms have an identical coord_mode distribution (pixel 14, normalized_1000 38) when they should differ. This applies uniformly to every model evaluated on that input.

gemma_31b_improved/t_plus_1_neg used the same input revision but had its 16 zero-length episodes re-run after applying commit ff747bd; the first-pass output is preserved in that cell rather than deleted. Whether those 16 episodes are the same 16 is unverified.

4. Older cells carry outcomes only

Cells produced before the artifact-retention change have no intermediate step records and no screenshots — episodes/<task>/ holds the episode outcome and nothing else. MANIFEST.json:status.artifacts reports per cell which of action_audit, agent_rollout, and replay_debug are present. Analyses that need step-level evidence must filter on those flags; there is no way to recover the missing artifacts from these runs.

5. MarkorTranscribeVideo fails structurally in the positive arm

actuation.py has no input_text branch, so the action dispatches as long_press(None, None). This reproduces across five models, occurs only in the t_plus_1_pos arm, and is unrelated to agent capability. The episode is retained and the denominator stays 52 — it is a harness limitation counted as a failure.

6. Five LLM-verified subset cells: header and rows disagree

gemini_flash_lv177, gemini_flash_lv254, guiowl15_8b_lv177, ma3_7b_lv177, and ui_tars_1_5_7b_lv177 were published between a denominator correction and a header fix. Their rows are correct; their result_count and success_rate header fields are from the pre-correction 312-entry state, and the success_rate field does not even equal success_count / result_count within the same file. Three different numbers are readable from one file:

Cell	Header	success_rate field	Recomputed from rows
guiowl15_8b_lv177	81/312	31.41%	45.76% (81/177)
gemini_flash_lv177	90/312	49.36%	50.85% (90/177)
gemini_flash_lv254	131/312	49.36%	51.57% (131/254)
ma3_7b_lv177	85/312	45.83%	48.02% (85/177)
ui_tars_1_5_7b_lv177	43/312	19.23%	24.29% (43/177)

score_tables.py uses the rows recomputation and --verify fails on these until the headers are corrected.

7. t_plus_1 exists as two different sets — check entry_set

There is a t_plus_1 condition with 65 entries, produced on a different machine, that is distinct from the 52-task t_plus_1_pos / t_plus_1_neg arms. The paper tables use the 52-task arms. Seven models (gui_owl_7b, ma3_7b, mai_ui_8b, qwen3vl_2b, qwen3vl_4b, qwen3vl_8b, ui_tars_7b_sft) have both, so a lookup by model and condition name alone will pick the wrong one. Split on MANIFEST.json:identity.entry_set, or on validation.json:expected_entries (65 vs 52).

8. Reproducibility caveats that are not defects in any single cell

• Rollout failures are counted in the denominator. An episode where the model emitted unparseable output is a failure, not an exclusion. This is intentional — excluding them would reward models that fail loudly — but it means a cell's score mixes "the agent tried and failed" with "the measurement failed." issues_by_type in each manifest lets you separate the two.
• Rollout failures re-run selectively would bias results. Where the cause is the model's own output (truncated tool-call JSON, hallucinated element indices), re-rolling until a cell passes advantages that cell. Those episodes are accepted as failures and the reason is recorded in accepted_rationale. Only harness and infrastructure failures were re-run.
• mobilerun_gemini_flash_3.0 does not honour the step budget. MobileRun runs its own agent loop at a flat 160 framework steps and ignores per-task recovery_max_steps (inference/android_world_agents.py:95). 23/65 episodes exceeded their task budget and 11 of those succeeded. Applying the standard rule gives 31/65 rather than the recorded 42/65. These cells carry step_budget_policy: mobilerun_flat_160; do not difference them against harness-enforced rows. For comparison, across the other 26 replay_65/t cells there were 568 budget-exhausted episodes and 0 successes.
• Serving reachability was observed differently from different hosts at the same time, and flipped within 13 minutes on one host. Reachability claims in the manifests are qualified by host and time for that reason.

---

Related repositories

Repository	Contents
Gyubeum/AndroidFlux_Failure_Recovery_Eval_Agent	Human-verified and LLM-verified inputs, VM snapshots, and the $t{+}1$ candidate-action inputs (human_verified_set/t_plus_1, llm_verified_set/t_plus_1)

Cells whose MANIFEST.json names Gyubeum/AndroidFlux_tplus1_inputs as their input were run from an earlier packaging of the human-verified $t{+}1$ inputs; that content is now human_verified_set/t_plus_1 of the input repository.

9. Seven cells carry header fields corrected during migration

Five LLM-verified subset cells (supplementary/llm_verified_subsets/t/*) were published with result_count and expected_entries of 312 and a success_rate copied from the parent 312-entry run; they now read 177 / 254 and the rate recomputed from their own rows. One human-verified cell (replay_65/t_minus_1/guiowl15_8b_think) had a success_rate field that did not equal success_count / result_count (both are 0). No success_count changed. The original values are kept in each cell's MANIFEST.json:corrections.

---

Citation

See the paper. If you use a single cell rather than a table, cite the cell by its MANIFEST.json:identity.cell_id and quote its known_issues — several cells in this repository are published because they are defective and a paper number rests on them.