AltimateAI / AltimateAI/altimate-code
[feat] Headless mode: commit best-guess answer at max-turns instead of meta-prose
- Linguagem predominante
- TypeScript
- Estrelas
- 811
- Forks
- 134
- Merge médio
- 3d 2h
- PRs com merge (30d)
- 50
Descrição
## Problem
In headless mode (`-p` / `--print`), when the agent hits `--max-turns`, the existing MAX_STEPS injection at \`packages/opencode/src/session/prompt.ts:881-888\` causes the model to emit multi-paragraph meta-prose instead of committing to an answer.
Concrete failure traces from a recent benchmark run (kimi-k2 and Claude Sonnet 4.6 both exhibit this):
> \"I have reached the maximum number of steps allowed for this task. Let me provide a summary of what has been accomplished and what remains to be done. Summary of Work Completed: 1. Successfully analyzed the question…\"
This happens even with explicit prompt instructions like \"always write SOMETHING to ANSWER before stopping\" — the MAX_STEPS injection overrides them.
In interactive mode this is fine — a human can react and ask follow-ups. In headless mode there's no human, so the program's final output is unparseable meta-prose. We measured ~5–10% of trials in a batch eval lost to this pattern.
## What it costs callers today
- Eval/benchmark frameworks (DAB, SWE-Bench, BIRD, internal evals): every team writes their own answer-extraction regex on top of altimate-code's prose, or counts those trials as fails.
- CI/CD pipelines: \`claude -p \"review this PR for security issues\"\` returns a status update instead of a partial review.
- Composable shell pipelines: \`git diff | claude -p \"summarize\" | tee summary.md\` is unreliable for long tasks.
- Predictable batch cost: trials that emit prose-instead-of-answer waste their full token budget.
## Proposed mechanism
A behaviour change scoped to **headless / `--print` mode only**:
1. Detect headless mode (CLI entry under \`packages/opencode/src/cli/\` or session config flag).
2. Branch the MAX_STEPS prompt:
- Interactive mode → keep existing text unchanged.
- Headless mode → use a different prompt that explicitly tells the model to commit a best-guess answer NOW.
Suggested headless prompt (refine to existing style):
\`\`\`
You have 1 turn left. Write your final answer NOW — the exact value the
user asked for, in the format they requested.
Do NOT summarize what you tried. Do NOT explain limitations. Do NOT write
meta-commentary about hitting the step limit.
Just emit the answer. If you're uncertain, emit your best guess anyway;
an uncertain answer is more useful than a meta-summary.
\`\`\`
**Optional stronger version (combine with the prompt fix):** on the final turn in headless mode, set \`tool_choice: required\` (or \`tool_choice: {type: \"tool\", name: \"return_answer\"}\` for tool-using flows). Mechanical guarantee on top of the prompt nudge.
**Optional pre-warning:** at \`step >= maxSteps - 2\`, inject a softer \"you're approaching your turn budget; start writing your answer if you haven't.\"
## What it enables
- Reliable batch / eval pipelines — recovers the ~5–10% of trials currently lost to meta-prose.
- CI/CD workflows that produce *actionable* output a build step can grep.
- Composable shell pipelines that work even under pressure.
- Predictable batch cost.
## Evidence we already have
In a sibling project we added a milder version of this instruction in our local agent prompt and saw the failure-mode flip on a hard query (0/4 → 1/4). Same model, same task, prompt-only delta.
## Implementation pointers
- Find headless detection: \`packages/opencode/src/cli/\`
- Find existing MAX_STEPS injection: \`packages/opencode/src/session/prompt.ts:881-888\`
- Likely a separate \`max-steps.txt\` (or similar) holding the prompt text — find and add a headless variant
- Estimated scope: ~10-30 LOC + tests
## Tests to add
- Headless-vs-interactive prompt selection unit test
- Snapshot of the new prompt text in headless mode
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.