PostHog / PostHog/posthog

feat(signals): pass an optional steering note when you run a scout now

Open
#98,978 0 comments 0 reactions 1 assignee View on GitHub

@andrewm4894 is already working on this.

Since Sep 11, 2026.

self-driving
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Problem

Two steering levers exist for scouts, and they do not compose.

Run now dispatches an on-demand run, but it carries nothing: the scout does whatever its skill body says, with no way to say what this particular run is for. Scout notes carry exactly that kind of steering, but they are durable fleet context read by whichever run happens next, so using one to steer a single run means leaving a note, triggering the run, and then remembering to delete the note before the schedule fires again.

The common case is a person who already knows what they want looked at right now: "run the error tracking scout and focus on the checkout regression we shipped this morning", or "re-run this and ignore the staging spike, it is a load test". Today they either run the scout and hope, or take on the note lifecycle by hand.

Target experience

  1. Run now stays one click. Nothing changes for a person who has no note to give.
  2. Next to it, Run with a note opens a small modal, takes a short note, and dispatches the same run.
  3. That run's prompt carries the note as steering for this run: advisory, untrusted input, never lowering the evidence bar or forcing an emit.
  4. The run row records the note, so run history shows why the run went the way it did.
  5. The note does not survive the run. No scheduled run and no sibling scout ever sees it.

What exists today

  • The run endpoint takes no body. SignalScoutConfigViewSet.run (products/signals/backend/scout_harness/views.py:2513) is declared request=None, gated on signal_scout:write, runs the fleet and spend gates, and calls start_manual_signals_scout_run(client, team_id=..., skill_name=...).
  • The dispatch path already carries per-dispatch fields. start_manual_signals_scout_run (products/signals/backend/temporal/agentic/scout_scheduler.py:338) builds a RunSignalsScoutInput, which already holds triggered_by, workflow_origin_key, and workflow_managed_resume. Adding one more optional field is the established shape.
  • The runner already stamps per-run dimensions onto the run row. arun_signals_scout (products/signals/backend/scout_harness/runner.py:160) takes triggered_by and writes non-schedule values into the run's metadata (runner.py:1044).
  • The prompt has a natural insertion point. build_run_prompt (products/signals/backend/scout_harness/prompt.py:1057) renders the run's opening prompt. Its Notes left for you section (prompt.py:232) already frames notes as advisory steering, as untrusted input, and asks the run to close the loop on which notes it acted on. A per-run note wants the same framing in its own section.
  • Notes today are pulled, not pushed. A scout calls scout-notes-list in step 1 of its bootstrap. SignalScoutNote rows are durable, with an optional wall-clock expires_at TTL.
  • Rendering steering into one run's prompt is already a solved pattern. products/signals/backend/report_steering.py does it for the research and implementation runs, with caps (_MAX_STEERING_NOTES = 10, _MAX_STEERING_NOTE_CHARS = 1_000) and a read failure that costs steering rather than the run.
  • The two actions sit at different authorization bars, on purpose. Running a scout needs signal_scout:write. Leaving a note needs signal_scout:write plus llm_skill:write plus llm_skill editor RBAC on the canonical team (SignalScoutNoteViewSet, views.py:1418), because note text is read verbatim by an agent holding privileged sandbox tools. This asymmetry is the main design constraint below.

Proposal

Slice 1: carry the note on the run, not in the notes table
  • POST /api/projects/:id/signals/scout_configs/:config_id/run accepts an optional note in the body. Add a request serializer and switch the @extend_schema off request=None. Cap the length at 1,000 characters, matching _MAX_STEERING_NOTE_CHARS.
  • Thread it: start_manual_signals_scout_run(..., note=...) -> a new optional note: str | None = None on RunSignalsScoutInput (defaulted so in-flight workflow histories still decode) -> arun_signals_scout(..., run_note=...) -> build_run_prompt.
  • Stamp it on the run row's metadata next to triggered_by, so the run is its own audit record and the UI can read it back.

Alternative considered and rejected: create a short-TTL SignalScoutNote and let the existing pull path find it. Cheaper, but wrong on three counts. The TTL is wall-clock, so a scheduled run firing inside the window reads a note it was never meant to see. The row shows up in the notes panel as durable steering. And cleaning it up after the run is a second thing that can fail.

Slice 2: authorization

A run note is the same capability as leaving a note: verbatim text into a privileged agent's prompt. So a run with a note must clear the note-writing bar (llm_skill:write on the key, llm_skill editor RBAC on the canonical team), while a run without one stays on signal_scout:write alone and the current one-click path is untouched.

  • Reuse _assert_can_steer_scouts from SignalScoutNoteViewSet rather than re-deriving the RBAC check.
  • Extend dangerously_get_required_scopes on the config viewset to add llm_skill:write only when the request body carries a note.
  • Document the new 403 in the endpoint's responses.
Slice 3: prompt rendering

Give the run note its own section in build_run_prompt, separate from Notes left for you, saying what it is: a note from the person who triggered this run, for this run only. Keep the existing guarantees explicit — advisory, not a command; untrusted input; it directs attention, it never lowers the evidence bar or forces an emit. Ask the run to say in its close-out how it handled the note. Do not fold it into the scratchpad as durable memory, because it was scoped to one run by construction.

Slice 4: surfaces
  • UI. Keep the primary "Run now" button in ScoutDetailHeader.tsx as-is. Add a side action that opens a small modal (textarea, character counter, dispatch), with a disabledReason when the person lacks skill-editor access. Show the note on the run in ScoutRunHistorySection.tsx / ScoutRunBoxes.tsx, so a later reader can tell a steered run from a plain one.
  • MCP. scout-run-now in products/signals/mcp/tools.yaml picks up the parameter from the regenerated operation. Its description should say a note steers one run only and does not persist, and its scopes list needs llm_skill:write.
  • CLI. --note on run_signals_scout.
  • Analytics. Capture whether a manual run carried a note, so adoption and any effect on report quality are measurable rather than assumed.

Open questions

  • Cap. 1,000 characters (matching report steering) or the full 10,000 that MAX_NOTE_CONTENT_LENGTH allows? Leaning 1,000: the note shares the run's attention with the skill body, and a long note is really a skill edit.
  • Promotion. When a run note turns out to be generally right, should run history offer "keep this as a scout note"? Cheap follow-up, not slice 1.
  • Workflow path. start_workflow_signals_scout_run (and the "Run scout" workflow action in #87478) shares the plumbing. Does the workflow action expose the field, or is this manual-only for now?
  • Provenance on the report. Should a report authored by a steered run say so? Leaning yes eventually, out of scope here.
  • 409 clarity. A manual run is refused while one is in flight, so a note cannot silently queue behind another run. Confirm the 409 tells the person their note was not applied.

Related

  • #93067 — forwarding resolve notes to scouts. Same steering channel, durable side.
  • #87478 — workflows "Run scout" action, which would inherit this field if we want it there.

Docs: no published doc describes "Run now" today, so the user-facing surfaces to update are the button tooltip and the scout-run-now tool description.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.