PostHog / PostHog/wizard

Sanitize wizard remarks and make remark parsing robust

Open
#812 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
197
Forks
51
Avg merge
1d 23h
Merged PRs (30d)
94

Description

Problem

The end-of-run "remark" is free-form model output that we extract from the agent's prose by string-matching a [WIZARD-REMARK] marker (AgentOutputSignals.remark() in src/lib/agent/output-signals.ts) and then ship straight to analytics:

const remark = signals.remark();
if (remark) analytics.capture(WIZARD_REMARK_EVENT_NAME, { remark });

Two problems with this:

1. It's not sanitized (security)

The remark is untrusted, model-generated text captured from output and sent to PostHog analytics with no sanitization. It can carry things we don't want to ingest verbatim: secrets/API keys the model echoed from the environment, file paths or PII, prompt-injection payloads, or control/escape characters. We should sanitize before capturing — strip control chars, redact secret-shaped tokens, cap length, and validate it looks like a single reflection line rather than arbitrary content.

2. Parsing it out of prose is flaky

Marker-scanning the agent's output stream is fragile. We've already hit failure modes:

  • The model echoes the ask itself instead of a real remark (fixed narrowly in #806 / the REMARK_INSTRUCTION echo check in remark()).
  • Only the first line after the marker is captured; multi-line or marker-mid-line output is dropped or truncated.
  • Any change to prose formatting can silently break extraction.

As noted in the thread: parsing wizard logs/output for remarks is inherently flaky and should be reworked.

Proposed direction

Move remarks off free-text prose parsing onto a structured channel — e.g. have the agent emit the remark via a dedicated tool call / structured output field instead of an inline marker, so it arrives as a typed value we don't have to regex out of the stream. Then run whatever we receive through a sanitization pass before analytics.capture.

Acceptance criteria

  • Remark content is sanitized (control chars stripped, secret-shaped tokens redacted, length-capped) before being captured to analytics.
  • Remark delivery no longer depends on regex-scanning prose — it comes through a structured/typed channel.
  • Echo-of-the-ask and empty/garbage remarks are reliably rejected.
  • Tests cover sanitization and the new extraction path.

Context: found while debugging the Windows skill-install issue; remarks broke (#806) and Vincent flagged the parsing approach as flaky and wanting a rework.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/lib/agent/output-signals.ts, including AgentOutputSignals.remark() and the REMARK_INSTRUCTION echo check, then trace the analytics.capture call for WIZARD_REMARK_EVENT_NAME. Review the existing remark flow and determine where the structured delivery and sanitization should be tested; done means typed remark delivery, rejected echoes and empty values, and sanitized analytics payloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
security, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.