Self-optimising workflows: suggestion loop tracking

Tracking issue for self-optimising workflows: PostHog's scout suggests a change to a workflow, a person approves it, and the change reaches people only through the normal publish. Design in the internal RFC requests-for-comments-internal#1108; this is its first phase.

Everything is behind the self-optimising-workflows flag, and off per workflow until the owner turns "Suggest improvements" on. Customer docs: Workflow suggestions, PostHog/posthog.com#19863, draft until rollout.

Goal

Close the loop a workflow owner never closes today: notice an email step performing badly, change it, and see whether the change helped. Suggestions only. No A/B testing, no auto-apply, and only PostHog's own scout files them; a person can only approve or reject.

The stack

Merge the whole stack with /trunk merge on the top PR. Each PR targets the branch below it so its diff shows one layer; nothing merges into a branch.

• #91708 WorkflowProposal: the record, tenant-scoped, with the migration
• #91709 the API: create, list, approve, reject, and the before/after outcome
• #91710 the workflow page: the Suggestions tab and the notice
• #82740 the MCP tools the scout calls (workflows-suggest, workflows-list-proposals)
• #91795 the per-workflow opt-in and the producer's work list
• #92252 signals-scout-workflows, the scout, plus per-version metrics and the review surfaces

Also open, extracted from the same work: #84065 (binds the app_metric_totals query filters, a pre-existing 500).

Decisions that matter for review

• One model holds a suggestion. It stores only the fields the change touches, who filed it, and its status through the queue. No second draft table. Fields exist only for what runs today; a new producer or surface adds columns when it arrives.
• Approving stages a draft, never live config. Approve writes the suggestion into the workflow's draft, the same move as restoring a revision. Publish ships it and records the version. Discarding the draft, restoring a revision, or approving another suggestion returns the earlier one to the queue, so publish cannot mark as applied something that never shipped.
• A suggestion carries only what it changes. Steps are keyed by id and merged field by field into the live step; an unrelated edit made while the suggestion waited survives approval. A suggestion is read against the version it names (base_version, required), so a step sent whole still merges as the fields that differ. Approve is refused (409) only when one of those fields moved to a value other than the proposed one; a draft edit hands an approved suggestion back only if it takes that change out of the draft. edges and variables replace the whole list, so for those any publish since counts. Create refuses a change publish would refuse.
• Provenance is server-derived. created_via comes from the request (a Signals run records self_driving), so the "Suggested by PostHog" label cannot be set by a caller. source_id names the run, so a retry returns the row it already made.
• Suggesting has its own scope, held only by the scout. hog_flow_proposal:write can file; publishing, editing and test-sending still need hog_flow:write, which no scout token carries. The scout declares the scope in its SKILL.md and the harness seeds it. It is not offered on personal API keys, but the server does not yet refuse it on one; making it server-minted only is listed under Later.
• The numbers a person judges are PostHog's, not the scout's. When a suggestion is filed, the server reads the step's own metrics at base_version and stores that reading beside the scout's evidence; the card shows the reading, labelled "Measured by PostHog", and says so when the scout's number disagrees. The scout's evidence still has to carry a unit, a denominator and counter-metrics, or create refuses it. Open and click rates divide by tracked sends; under 20 observations the card says so instead of presenting a result.
• Per-version metrics are how "did it help" is answered. metrics/totals?version=<n> reads one published version's series. A version is not a subset of the unversioned read (batch and broadcast runs key on the run), so versions are compared with each other. The scout waits two days of opens before judging a version.
• The opt-in is a row per workflow; turning it off keeps the row. Only a live workflow can be opted in or suggested against (a draft has no sends, an archived one is done), so the Suggestions tab stays on every saved workflow for discoverability, but its switch is disabled with "Suggestions need a live workflow. Enable it first." until the workflow is active. The server refuses a suggestion for a workflow that is off; the scout's work list is the workflows that are on. Its read scope still covers the whole project, so "reads only opted-in workflows" is a skill rule, not a server one. "Tried it and turned it off" is a rollout question. Cadence lives with the scout config, not on the row.
• The scout files no inbox report and emits no signal. An actionable report can dispatch a code run and open a billable pull request; this change is configuration, not code. Its only output is the suggestion on the workflow.

Rollout

• Before merging, add signals-scout-workflows to the signals-scout flag's withheld_skills, so the scout does not start on every Signals-enrolled team when the stack lands; release it per team as the workflows flag rolls out.
• Billing: Signals bills per implemented report; this scout files none, so today it costs the customer nothing and PostHog inference. RFC open question 3. Until decided, no surface says it uses credits.
• A run on a cloud project on the signals-scout allowlist, and a run the coordinator's own schedule fires. Everything so far ran in a local sandbox against the local stack.

After it ships, adoption reads from the hog_flow_optimisation_enabled / _disabled and hog_flow_proposal_approved / _rejected events, and the suggestions themselves from the WorkflowProposal rows.

Later

• Make hog_flow_proposal:write server-minted only (an internal scope object), so no personal key or OAuth app can file a suggestion, not just none is offered one. It needs the scout grant path to allow an internal object for this one scout.
• Show the version's age on the card ("v5, live 3 hours") from its revision row, so a suggestion filed against a young version is visible as such rather than trusted to the scout's two-day rule.
• Desktop delivery: a report kind the harness pins as non-implementable, carrying the proposal id with approve and reject actions. The model stays the record.
• Say something when the scout cannot judge a workflow: tracking off, too few sends, metrics missing. Today that goes only to its scratchpad and the run record, so nobody hears that the thing blocking a suggestion is fixable.
• Notify when a suggestion lands, rather than waiting for someone to open the workflow. In-app first, through the notifications facade, to everyone who can edit the workflow. Desktop delivery above is the step after that.
• Unsubscribe rate as a counter-metric, once something emits email_unsubscribed.
• Engagement splits by version only for sends made after #91487; older versions read n=0, which the sample floor labels.
• The card charts a metric per version, not two arms, so a move is still not proof that the suggestion caused it. Deciding a winner is the RFC's A/B step.
• Let the scout name the metric it optimises for. The outcome read hardcodes the open rate today; it would read evidence.metric instead, which then needs validating against the metrics we can actually measure.
• Conversions as the headline metric, once conversion rates are stored per version and step. That is a metrics change rather than a proposal one.
• One metric over time for the whole workflow, with a marker on every published version rather than only the ones around a suggestion, so a person can see what drove what.
• An enable action on the workflow itself for workflows nobody opted in yet. The tab and the list already say when it is on; what is missing is the invitation when it is off.
• More than one suggestion at a time. The model allows it, but approving a second one rebuilds the draft from live content and drops the first. Merging onto the staged draft instead would let two suggestions on different steps stack into one publish.
• Feed the outcome back into the next scout run, so a change that made things worse can be suggested as a revert. Andy's report checks are the natural home for this.
• Broadcasts: their per-version metrics already aggregate across batch runs, but a one-shot send has nothing to improve, so the work list should skip them or take only recurring ones.

Not in scope

The RFC's later phases: A/B testing between variants, auto-apply of winners, personalisation, structural changes to a workflow, and cross-workflow analysis.