PostHog / PostHog/posthog

Scout `edit_report` mutates any team report with no safety judge, and re-runs autostart

Open
#73,689 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
7h 27m
Merged PRs (30d)
222

Description

Problem

emit_report runs the safety judge before persisting, and the resulting status comes from safety × actionability. edit_report does not. products/signals/backend/scout_harness/tools/report.py validates inputs and goes straight to _do_edit_report; there is no equivalent of the judgement.safety path emit_report takes.

The edit path is also broad. Per the report-channel contract, edit_report can target any of the team's inbox reports, not just ones the scout authored, and its own docstring notes that setting suggested_reviewers "re-runs autostart so a report missing a qualifying reviewer can open a draft PR".

So an edit can rewrite a human- or pipeline-authored report's title and summary, change who it routes to, and re-trigger a draft PR, with no safety evaluation of the content being written.

Why it matters now

Scouts read untrusted, user-authored content by design. All 28 canonical scouts declare edit_report in allowed_tools (the authoring-scouts skill tells authors every scout should), and several read content an outside party controls:

Scout Untrusted input
signals-scout-conversations support ticket threads
signals-scout-skills-store user-authored skill bodies
signals-scout-mcp-tool-calls client-submitted telemetry, error text
signals-scout-error-tracking exception messages
signals-scout-tasks task titles and descriptions (prompts people wrote for agents)

The scout prompts and skill bodies all carry data-not-instructions guidance, and that is the current mitigation. It is prompt-level, so it holds right up until it doesn't — and the blast radius on the other side is a mutated report on a shared surface plus a possible autostarted PR.

This surfaced in review of #73685. That PR hardened its own body (no instruction found in task data may reach the report tools), but the exposure is shared and can't be closed scout by scout — dropping the tool from one scout leaves 27 others and breaks the dedupe-by-edit discipline the fleet is built on.

Candidate fixes

  1. Run the safety judge on edit content, same as emit_report, over title / summary / append_note. Closest to the existing model and the smallest conceptual change.
  2. Narrow the default scope. Let a scout edit reports it authored (or that its own signals back) and require something explicit beyond that. Most legitimate scout edits are to their own live reports; the report:<domain>:<entity> scratchpad pointer already encodes that relationship.
  3. Split reviewer mutation out of the general edit path, so the autostart re-trigger is not reachable by the same call that rewrites prose.

1 and 2 compose and are probably the right pair.

Notes

Worth deciding explicitly whether append_note should be treated differently from a title / summary rewrite. A note is additive and audit-friendly (the contract already recommends it over rewriting someone else's report), so it may warrant a lighter gate than an in-place overwrite of a human-authored headline.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in products/signals/backend/scout_harness/tools/report.py, comparing edit_report and _do_edit_report with the safety-judge path used by emit_report. Review the report-channel contract and the three candidate fixes, including how append_note, reviewer changes, and autostart should behave. Done means the chosen safety and scope rules are implemented without leaving the shared edit path exposed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.