anthropics / anthropics/claude-code

`/ultrareview` results become permanently unrecoverable once the task ID expires

Open
#88,606 0 comments 0 reactions 0 assignees View on GitHub
area:cost area:skills bug has repro
Dominant language
Python
Stars
145k
Forks
23.1k
PR merge metrics
PR metrics pending

Description

# `/ultrareview` results become permanently unrecoverable once the task ID expires

## Summary

`/ultrareview` is a paid/quota-limited feature (e.g. "Free ultrareview 2 of 3") that runs an expensive cloud review. Once it completes, its findings are delivered as a `` into the *assistant's* model context only — the user never sees this raw payload directly; the user's only view of the results is the assistant's own prose reproduction of them in the chat. There is no user-facing, durable way to retrieve the raw output at all: the tracking URL shows nothing, and `TaskOutput` stops resolving the task ID well within the same session (so even the assistant can't re-fetch it). The only way to get a review's data again is to re-run `/ultrareview`, which consumes another one of the limited runs and produces a fresh (not identical) analysis.

Because this feature specifically costs real quota per invocation, having no user-facing access to its raw output — only a model-mediated retelling — is a bigger problem than it would be for a free/cheap background task. You can't just "run it again" for free to recover the data, and you can't get the actual results without trusting the assistant's regeneration of them.

## Repro

1. Ran `/ultrareview` on a branch (billed against a limited quota — "Free ultrareview 2 of 3"). It launched a cloud review and returned a task ID plus a tracking URL (`https://claude.ai/code/session_...`).
2. The review completed; a `` delivered the full JSON findings payload into the assistant's context — not anywhere the user could see it directly. The user's only visibility into the results was the assistant retyping/summarizing them in chat.
3. User visited the tracking URL afterward — no output was shown there, so there was no user-facing way to see the raw results either.
4. Later in the same session, the assistant called `TaskOutput` with the same task ID to try to re-fetch the raw payload on the user's behalf:
```
TaskOutput({ task_id: "rccpjvyqu", block: false })
→ Error: No task found with ID: rccpjvyqu
```

## Expected

Given `/ultrareview` charges against a limited quota per run, the user should have some direct, durable, user-facing way to see the actual results — not only the assistant's paraphrase of them. Either:
- the tracking URL should actually show the completed review's raw output to the user, or
- the raw findings should be surfaced to the user directly in the conversation (e.g. as a file, artifact, or rendered block) at delivery time, not only handed to the assistant's context, or
- `TaskOutput` (or an equivalent) should still retrieve the completed review's result for a reasonable retention window so it can be recovered later, or
- the task notification should include a stable output file/link that persists independently of the ephemeral task ID.

## Actual

- The user never had direct access to the raw findings at all — only the assistant's own chat reproduction of them.
- The tracking URL showed nothing.
- The task ID was already unresolvable via `TaskOutput` well within the same conversation/session, so not even the assistant could recover the original data afterward.

## Impact

For a feature that costs real, limited quota per run, users can't get a guaranteed-exact copy of the findings at all — they only ever see the assistant's regenerated retelling of them, with no fallback to the actual source once the task ID expires. Re-running the review to "get the data" burns another one of a limited number of paid/quota-gated runs.

This also locks in whichever model happened to be driving the session at delivery time. In this case the session was running Sonnet, so the retelling of an expensive, quota-gated review is permanently capped at Sonnet-quality synthesis — there's no way to later have a higher-tier model (Opus, Fable) re-read the same raw findings and produce a better analysis, because the raw data no longer exists anywhere retrievable. The model tier active when the task notification lands becomes a permanent, unchangeable property of the result.

## Related

- #87285 — a different failure mode on the same theme: there, the `task-notification` never arrives at all and the browser session 403s. Here, the notification *does* arrive (into the assistant's context) but still can't be recovered afterward by anyone.
- #63331 — docs describe a `/tasks` command and `claude ultrareview --json` subcommand for reading findings after the fact; neither exists in the shipped CLI.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the /ultrareview flow, task-notification delivery, TaskOutput lookup, and tracking URL behavior described in the report. Done means completed review findings remain directly and durably accessible to the user without rerunning the quota-limited review or relying only on an assistant paraphrase.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.