AltimateAI / AltimateAI/altimate-code

[feat] Headless mode: commit best-guess answer at max-turns instead of meta-prose

Abierto
#759 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement priority:medium ux
Lenguaje dominante
TypeScript
Estrellas
811
Forks
134
Merge medio
3 d 2 h
PR fusionados (30 d)
50

Descripción

## Problem

In headless mode (`-p` / `--print`), when the agent hits `--max-turns`, the existing MAX_STEPS injection at \`packages/opencode/src/session/prompt.ts:881-888\` causes the model to emit multi-paragraph meta-prose instead of committing to an answer.

Concrete failure traces from a recent benchmark run (kimi-k2 and Claude Sonnet 4.6 both exhibit this):

> \"I have reached the maximum number of steps allowed for this task. Let me provide a summary of what has been accomplished and what remains to be done. Summary of Work Completed: 1. Successfully analyzed the question…\"

This happens even with explicit prompt instructions like \"always write SOMETHING to ANSWER before stopping\" — the MAX_STEPS injection overrides them.

In interactive mode this is fine — a human can react and ask follow-ups. In headless mode there's no human, so the program's final output is unparseable meta-prose. We measured ~5–10% of trials in a batch eval lost to this pattern.

## What it costs callers today

- Eval/benchmark frameworks (DAB, SWE-Bench, BIRD, internal evals): every team writes their own answer-extraction regex on top of altimate-code's prose, or counts those trials as fails.
- CI/CD pipelines: \`claude -p \"review this PR for security issues\"\` returns a status update instead of a partial review.
- Composable shell pipelines: \`git diff | claude -p \"summarize\" | tee summary.md\` is unreliable for long tasks.
- Predictable batch cost: trials that emit prose-instead-of-answer waste their full token budget.

## Proposed mechanism

A behaviour change scoped to **headless / `--print` mode only**:

1. Detect headless mode (CLI entry under \`packages/opencode/src/cli/\` or session config flag).
2. Branch the MAX_STEPS prompt:
- Interactive mode → keep existing text unchanged.
- Headless mode → use a different prompt that explicitly tells the model to commit a best-guess answer NOW.

Suggested headless prompt (refine to existing style):

\`\`\`
You have 1 turn left. Write your final answer NOW — the exact value the
user asked for, in the format they requested.

Do NOT summarize what you tried. Do NOT explain limitations. Do NOT write
meta-commentary about hitting the step limit.

Just emit the answer. If you're uncertain, emit your best guess anyway;
an uncertain answer is more useful than a meta-summary.
\`\`\`

**Optional stronger version (combine with the prompt fix):** on the final turn in headless mode, set \`tool_choice: required\` (or \`tool_choice: {type: \"tool\", name: \"return_answer\"}\` for tool-using flows). Mechanical guarantee on top of the prompt nudge.

**Optional pre-warning:** at \`step >= maxSteps - 2\`, inject a softer \"you're approaching your turn budget; start writing your answer if you haven't.\"

## What it enables

- Reliable batch / eval pipelines — recovers the ~5–10% of trials currently lost to meta-prose.
- CI/CD workflows that produce *actionable* output a build step can grep.
- Composable shell pipelines that work even under pressure.
- Predictable batch cost.

## Evidence we already have

In a sibling project we added a milder version of this instruction in our local agent prompt and saw the failure-mode flip on a hard query (0/4 → 1/4). Same model, same task, prompt-only delta.

## Implementation pointers

- Find headless detection: \`packages/opencode/src/cli/\`
- Find existing MAX_STEPS injection: \`packages/opencode/src/session/prompt.ts:881-888\`
- Likely a separate \`max-steps.txt\` (or similar) holding the prompt text — find and add a headless variant
- Estimated scope: ~10-30 LOC + tests

## Tests to add

- Headless-vs-interactive prompt selection unit test
- Snapshot of the new prompt text in headless mode

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.