multica-ai / multica-ai/multica

[Feature]: Dry-run mode for autopilots and tasks — render the prompt without spending tokens

Open
#4,100 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
50k
Forks
6.5k
Avg merge
1d 36m
Merged PRs (30d)
500

Description

### What do you want and why?

A `--dry-run` mode (or equivalent task-level flag) that lets an operator **rehearse** an autopilot or one-off task without actually invoking the agent CLI, so the cost and shape of "what would have happened" is visible *before* spending tokens.

### The current gap

When you wire up a new autopilot — cron trigger, agent assignment, prompt template — there's no way to ask the platform "if I turned this on, what would the next 24h look like?" The first real signal you get is the agent CLI actually running, billing tokens, and writing to the issue. If the prompt template references an issue field that's empty for most issues, or the trigger fires more often than intended, you find out by paying for it.

Combined with how usage data surfaces — task_usage is recorded on completion and rolled up hourly (`taskusagebackfill`) — by the time you see the burn on the dashboard, it's already an hour stale and several runs deep.

This shows up in three places where it would meaningfully change operator behavior:

1. **Autopilot validation.** "I just created a trigger that fires every weekday at 9am. Show me the actual rendered prompt for next Monday's run, without spending money."
2. **Spend forecasting.** Pairs naturally with the spend-cap proposal in #4076: "if I set this autopilot live, the projected daily cost is ~$X / ~Y tokens / ~Z turns at the configured `--max-turns`."
3. **Tightening prompt templates.** Right now the round-trip is: edit prompt → wait for next fire → read transcript. A dry-run shortcut is "render the prompt for these last 5 issues" and read them side-by-side.

### Proposal

Add `ExecOptions.DryRun bool` in `server/pkg/agent/agent.go`. When set, the agent backends (claude.go, codex.go, etc.) **build the full command line and prompt exactly as production**, but instead of spawning the CLI, return a structured `DryRunResult{ RenderedPrompt, CommandArgs, EnvKeys, EstimatedInputTokens }` to the daemon. The daemon writes this into the task's `result` JSON with a `dry_run: true` marker. UI badges the task as a rehearsal.

Surface points:

- CLI: `multica autopilot trigger --dry-run` — useful for ops.
- CLI: `multica issue create … --dry-run` — useful for prompt iteration.
- API: a `dry_run` field on the trigger / chat-message endpoints.
- Optional: `multica autopilot forecast --window 7d` — compute projected runs from the cron expression × estimated tokens per run.

### Scope discipline

- No new schema. `result` JSON is already opaque; the marker lives there.
- No changes to the agent CLI integrations beyond a `DryRun` short-circuit in the spawn step.
- Token *estimate* uses a coarse `len(rendered_prompt) / 4` heuristic — exact accounting requires hitting the model, which defeats the purpose.
- Skill discovery and runtime config injection still run (so dry-run faithfully shows what an executor would actually receive), they're just not followed by a real model call.

### Companion PR

I'm posting a draft implementation in a follow-up PR (linked when ready). Wanted to file this issue first so the design conversation has a home that isn't a 400-line diff.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start in server/pkg/agent/agent.go and inspect how ExecOptions reaches the agent backends in claude.go, codex.go, and related files. Trace the autopilot trigger and issue-create CLI entry points plus the API endpoints named in the issue. Done means production-equivalent prompts and command inputs are returned without spawning the agent CLI, with the dry-run result recorded in task result JSON.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.