AltimateAI / AltimateAI/altimate-code

[feat] Headless mode: commit best-guess answer at max-turns instead of meta-prose

Aberta
#759 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement priority:medium ux
Linguagem predominante
TypeScript
Estrelas
811
Forks
134
Merge médio
3d 2h
PRs com merge (30d)
50

Descrição

## Problem

In headless mode (`-p` / `--print`), when the agent hits `--max-turns`, the existing MAX_STEPS injection at \`packages/opencode/src/session/prompt.ts:881-888\` causes the model to emit multi-paragraph meta-prose instead of committing to an answer.

Concrete failure traces from a recent benchmark run (kimi-k2 and Claude Sonnet 4.6 both exhibit this):

> \"I have reached the maximum number of steps allowed for this task. Let me provide a summary of what has been accomplished and what remains to be done. Summary of Work Completed: 1. Successfully analyzed the question…\"

This happens even with explicit prompt instructions like \"always write SOMETHING to ANSWER before stopping\" — the MAX_STEPS injection overrides them.

In interactive mode this is fine — a human can react and ask follow-ups. In headless mode there's no human, so the program's final output is unparseable meta-prose. We measured ~5–10% of trials in a batch eval lost to this pattern.

## What it costs callers today

- Eval/benchmark frameworks (DAB, SWE-Bench, BIRD, internal evals): every team writes their own answer-extraction regex on top of altimate-code's prose, or counts those trials as fails.
- CI/CD pipelines: \`claude -p \"review this PR for security issues\"\` returns a status update instead of a partial review.
- Composable shell pipelines: \`git diff | claude -p \"summarize\" | tee summary.md\` is unreliable for long tasks.
- Predictable batch cost: trials that emit prose-instead-of-answer waste their full token budget.

## Proposed mechanism

A behaviour change scoped to **headless / `--print` mode only**:

1. Detect headless mode (CLI entry under \`packages/opencode/src/cli/\` or session config flag).
2. Branch the MAX_STEPS prompt:
- Interactive mode → keep existing text unchanged.
- Headless mode → use a different prompt that explicitly tells the model to commit a best-guess answer NOW.

Suggested headless prompt (refine to existing style):

\`\`\`
You have 1 turn left. Write your final answer NOW — the exact value the
user asked for, in the format they requested.

Do NOT summarize what you tried. Do NOT explain limitations. Do NOT write
meta-commentary about hitting the step limit.

Just emit the answer. If you're uncertain, emit your best guess anyway;
an uncertain answer is more useful than a meta-summary.
\`\`\`

**Optional stronger version (combine with the prompt fix):** on the final turn in headless mode, set \`tool_choice: required\` (or \`tool_choice: {type: \"tool\", name: \"return_answer\"}\` for tool-using flows). Mechanical guarantee on top of the prompt nudge.

**Optional pre-warning:** at \`step >= maxSteps - 2\`, inject a softer \"you're approaching your turn budget; start writing your answer if you haven't.\"

## What it enables

- Reliable batch / eval pipelines — recovers the ~5–10% of trials currently lost to meta-prose.
- CI/CD workflows that produce *actionable* output a build step can grep.
- Composable shell pipelines that work even under pressure.
- Predictable batch cost.

## Evidence we already have

In a sibling project we added a milder version of this instruction in our local agent prompt and saw the failure-mode flip on a hard query (0/4 → 1/4). Same model, same task, prompt-only delta.

## Implementation pointers

- Find headless detection: \`packages/opencode/src/cli/\`
- Find existing MAX_STEPS injection: \`packages/opencode/src/session/prompt.ts:881-888\`
- Likely a separate \`max-steps.txt\` (or similar) holding the prompt text — find and add a headless variant
- Estimated scope: ~10-30 LOC + tests

## Tests to add

- Headless-vs-interactive prompt selection unit test
- Snapshot of the new prompt text in headless mode

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.