Fix terminal evaluation recovery and SFT data filtering for the KDA checkpoint

Codex · [Codex task](codex://threads/01a092b0-dc47-7252-9d20-120e3f819471)

Pradeep’s continued-SFT KDA checkpoint scores **1/100 on TBLite and 0/89 on Terminal Bench 2.1**. Inspection of all 189 saved trajectories found failed training demonstrations, long no-action reasoning loops, recovery prompts followed by premature submission, and two harness bugs. Fix these before deciding whether more training is needed; their score improvements have **not** been measured in an A/B.

Parent checkpoint: [1-epoch-dolci-think (#1859)](https://github.com/allenai/open-instruct/issues/1859).

Runs:

1. TBLite, pass@1 1%: [Beaker](https://beaker.org/ex/01M211D9Q8VCWYCMRRX1ED0QC6)
2. Terminal Bench 2.1, pass@1 0%: [Beaker](https://beaker.org/ex/01M20WJVQBQ8GEGJ0D9CD525V3)

## Run provenance and outcomes

Both evaluate `/weka/oe-adapt-default/allennlp/deletable_checkpoint_states/ecppxpon/hf_step604`, continued from the durable `step23607` supplied above. Training: one epoch, 604 steps, LR 2.5e-5, seq 65,536; W&B run `c08ewfs2`. Serving: vLLM 0.19.1, `qwen3_xml` + `olmo3` parsers, 65,536 context, two data-parallel replicas and eight concurrent trials; tmax revision `595caabfeaec`.

| Mutually exclusive trial ending | TBLite | TB 2.1 |
|---|---:|---:|
| Agent wall-clock timeout | 50 | 40 |
| Submit command executed | 40 | 39 |
| Context limit reached | 6 | 7 |
| 64-step limit reached | 3 | 2 |
| Individual bash command timeout, 120 seconds | 1 | 1 |
| Total | 100 | 89 |

Both Beaker jobs exited successfully. Trial outcomes and verifier results explain the scores.

## Findings

**1. Failed demonstrations were included in SFT.** The matching cache, `/weka/oe-adapt-default/allennlp/deletable_open_instruct_dataset_cache/numpy_sft/bb1d94703a-6068a350`, contains 633M tokens from SimFC and TMAX. Its manifest records TMAX revision `00b28cd071976a2cadcde3f5c5201f772eba6d6c`, whose [default config is `all`](https://huggingface.co/datasets/allenai/tmax-sft-glm-52/blob/00b28cd071976a2cadcde3f5c5201f772eba6d6c/README.md): 16,728 trajectories, versus 10,029 in `successful`. The cache confirms all 16,728 were loaded and ordinary tokenization/filtering retained 16,037. Even if all 691 dropped rows were unsuccessful, **at least 6,008 unsuccessful trajectories remained**. TMAX supplies 53.2% of trainable tokens. The exact retained success count was not recomputed.

**2. Long reasoning without actions consumes much of the budget.** With the exported tokenizer, at least **47 TBLite tasks and 44 TB tasks** contain a completed no-tool reply retokenizing to 16,380–16,384 tokens. These counts overlap the timeouts above. The 114 near-cap replies consume about **57% of logged completed LLM-call latency**, summed across concurrent requests. Timed-out in-flight replies are absent from saved trajectories.

**3. Recovery frequently leads directly to submission.** After a missing tool call, the harness says:

> If you want to end the task, please issue the command `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`

Then **37/40 TBLite submissions and 38/39 TB submissions occur immediately after the correction**. Only one submitted task passes. This is an observed sequence; whether the wording causes the exits needs an A/B.

For `bash-log-processor-fix`, the model reads files, produces two 16K-token no-action replies, and submits without editing the requested script. The final sequence is:

```text
Model: [second ~16K-token discussion; no bash call]
Harness: [format error, including the submit-command suggestion]
Model: bash(command="echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT")
Harness: [ends the agent attempt and runs the verifier]
Verifier: processed.log was not created. Score: 0.0
```

`echo` prints a marker; the [agent recognizes it and breaks the action loop](https://github.com/shatu/tmax/blob/595caabfeaec/Vanillux2Agent/agent.py#L211). The verifier grades the files/environment produced so far. Submission does not establish completion or correctness.

**4. Context budgeting and multiple-call handling are broken.** The agent always requests 16,384 output tokens, so vLLM rejects prompts above 49,152 tokens; the agent catches the exception and stops. This accounts for the 13 context exits. Separately, the [parser takes only `tool_calls[0]`](https://github.com/shatu/tmax/blob/595caabfeaec/rl_data/generator/sample_solutions.py#L131), while the agent retains the entire assistant message. **43 additional calls across 21 tasks** receive no execution/result.

## Proposed follow-up

- [ ] Train on `load_dataset("allenai/tmax-sft-glm-52", "successful", split="train")`. The inspected mixer does not expose HF config selection; a local Parquet export is a supported input route.
- [ ] Remove the submit example from format-error recovery. Request the next concrete action, e.g. “Your response contained no bash tool call. Take one concrete action toward completing the task. Describing a solution does not create or modify the required files.” Keep normal finishing instructions in the original task prompt.
- [ ] Pair that recovery change with a 4K per-action completion cap as a diagnostic. Set the **agent** limit, e.g. tmax runner `EXTRA_AGENT_KWARGS=max_tokens=4096`; its explicit request overrides the server default. A lower cap alone may just truncate the same loop earlier.
- [ ] Budget output against actual rendered prompt length and compact history when necessary.
- [ ] Execute all returned tool calls with matching responses, or explicitly reject multiple calls and repair the history; avoid silently dropping them.
- [ ] Run a controlled small-task comparison before another full benchmark or increasing training duration. Track score, no-action cap hits, timeout rate and recovery-to-submit transitions.

## Checks and limits

Cached training and exported evaluation templates are byte-identical; both tool smoke tests passed, and 2,765 tool-result messages were recorded. No literal `` blocks were stranded in saved assistant content. Only three cached full-window rows ended on trainable non-EOS tokens, making truncation-tail repair a low-priority explanation here.

Real capability errors remain: `kv-store-grpc` passes 5/7 verifier tests but declares request field `val` where `value` is required. These traces do not distinguish looping inherited from the parent checkpoint from behavior intensified by continued SFT.

Three local diagnostic tests and Ruff passed: parser behavior, reconciliation of all trial endings/context errors, and the 75 recovery-to-submit transitions. No new inference/training or numerical export-parity check was run. No new GPU jobs were launched.