PostHog / PostHog/posthog

Scouts: remove the per-scout emit (dry-run) setting

Open
#89,151 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature/signals self-driving
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
7h 27m
Merged PRs (30d)
222

Description

SignalScoutConfig.emit is a per-scout dry-run posture, defaulting to on. A scout should always be able to write and edit reports, so the setting has no job left to do. It is an early design relic and should come out.

Worth stating plainly: #88605 just added this toggle to the web settings form, because it was reachable through the API but not the UI. That was the right fix for the inconsistency it found, and it makes the setting's real cost easier to see. This issue removes the setting and the toggle with it. Closed #88966 as the issue behind that PR.

Why remove it rather than keep it

Three things make it worse than redundant.

It blocks the report channel, not just signals. _preflight_emit_gates in products/signals/backend/scout_harness/tools/emit.py:510 is shared by emit_finding, emit_report, and record_structured_output. A dry-run scout cannot author or edit a report at all. That contradicts what a scout is for.

It is not a preview mode, it is a shredder. Dispatch only filters on enabled (config_registry.py:58), so a dry-run scout runs in full and costs a full agent run. Then _gate_skip_result (report.py:326) returns report_id=None and persists nothing. The report the agent just wrote is discarded. Nobody can read what it would have said. So dry-run buys you the spend of a live scout and the output of a disabled one, and enabled=false already covers "do not run this".

It forces special cases across the harness. The inactivity sweep filters to emit=True and carries a comment explaining that a dry-run scout's silence means nothing (inactivity.py:185). structured_output.py:381 raises rather than record. runner.py:700 strips the schema snapshot and runner.py:973 withholds a prompt section. report.py:769 keeps a set of skip reasons that must not fan out to customers. Each is a branch that exists only because a scout might be configured to throw its work away.

What a person sees afterwards

  • No mode control in scout settings or the create form, in the web app or the desktop app.
  • No "Dry run" badge in the roster, the summary row, or the scout page. A scout is on or off.
  • A scout that runs writes its findings and reports. There is no configuration in which it runs and discards them.
  • The gates that stop output for a real reason still stop it, and still say why.

Removal surface

Backend:

  • Drop the field on SignalScoutConfig (models.py:1404) and the doc comments around it, including the structured_output_schema comment that says the channel requires emit.
  • _preflight_emit_gates: drop the scout_emit_disabled branch and its EMIT_SKIP_REMEDIATION entry, and remove it from _INACTIVE_SKIP_REASONS.
  • structured_output.py: drop the if not config.emit raise in _reserve_capacity, and the dry-run clause in InvalidStructuredOutputError's docstring.
  • runner.py: both config.emit conditionals become unconditional on the schema being set.
  • inactivity.py: drop the emit=True filter and its comment.
  • prompt.py:250: the fleet roster line telling other scouts that a dry-run scout's silence is meaningless.
  • Serializers: the field on the read, update, create-options, and fleet-entry serializers, plus the caveat in EmitEligibilitySerializer.can_emit's help text.
  • Views: the emit mentions in the list, create, update, and destroy endpoint descriptions.
  • admin.py:95,102: list_display and list_filter. These break at import if the field goes and admin does not, so they belong in the same change.

Frontend, web and desktop:

  • Web: ScoutConfigControls.tsx, ScoutCreateModal.tsx, ScoutBadges.tsx, ScoutSummaryRow.tsx, ScoutsRoster.tsx dry-run count, scoutGroups.ts, InboxWaitingForWork.tsx, scoutCreateModalLogic.ts, mocks and tests.
  • Desktop: features/scouts/components/ScoutConfigControls.tsx mode select, ScoutBadges.tsx, ScoutsFleetSection.tsx dry-run count, useScoutConfigMutations.ts tracked settings, the hand-rolled ScoutConfig.emit type in api-client/src/posthog-client.ts, and the stories.
  • The desktop app ships on its own cadence, so an older build will keep sending emit in its PATCH body for a while. DRF ignores unknown fields, so those writes become no-ops and the scout stays live. That is the right way for it to fail, but confirm it rather than assume it.

Analytics: setting on the desktop config-changed event drops an enum member, and the detail-viewed event drops a property on both clients. Historical events stay readable, so this is deletion, not a rename.

Docs and generated files: the run-posture sections in authoring-scouts and exploring-scouts, the harness CLAUDE.md, the field mentions in products/signals/mcp/tools.yaml, and a hogli build:openapi run for the generated TypeScript.

Do not remove these

emit is overloaded in this codebase, and three neighbors read like the same thing:

  • The other three preflight gates: scout_config_missing, ai_processing_not_approved, source_disabled. These are the fail-closed path and the org consent gate. _preflight_emit_gates keeps its shape, minus one branch.
  • can_emit in skill_loader.py and the project profile. That is whether a skill was granted the emit_report tool, which is a different mechanism.
  • The scout-emit-signal and scout-emit-report MCP tool names, and the emitted_count / emitted_finding_ids / emitted_report_ids run tallies. Those name the act of emitting, not the setting.

Migration

The column is a plain unindexed boolean on a product table, with no constraint, no index, and no ClickHouse mirror. It is not on a hot table. Two phases, per /django-migrations:

  1. Remove every reference, then a state-only removal wrapped in SeparateDatabaseAndState. Deploy. A Django SELECT lists every model field, so dropping the column while any running code still expects it breaks reads mid-deploy.
  2. DROP COLUMN in a later migration, once the first has been out for a full deploy cycle.

One decision for whoever picks this up

A handful of scouts in PostHog's own project are dry-run today, and they look like custom ones still being tuned. When the setting goes, they either start posting to the inbox or they stop.

Suggested default: disable them and tell their owners. From the inbox's point of view disabled and dry-run already look identical, disabled stops paying for discarded output, and the owner can turn it back on in one click. Flipping them live instead means someone gets output they never asked for from a scout they thought was still in staging. Worth a look at whether any enrolled team outside our own project is in the same position before choosing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing SignalScoutConfig.emit from models.py through emit.py, structured_output.py, runner.py, serializers, views, and admin.py, then review the web and desktop files listed in the issue. Follow /django-migrations for the two-phase field removal and run hogli build:openapi. Done means the setting, controls, badges, analytics references, docs, and generated files are removed while real output gates remain intact and legacy desktop PATCHes are harmless.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python, react, typescript
Domain
backend, databases, desktop, documentation, frontend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.