Sanitize wizard remarks and make remark parsing robust
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 197
- Forks
- 51
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 94
Description
Problem
The end-of-run "remark" is free-form model output that we extract from the agent's prose by string-matching a [WIZARD-REMARK] marker (AgentOutputSignals.remark() in src/lib/agent/output-signals.ts) and then ship straight to analytics:
const remark = signals.remark();
if (remark) analytics.capture(WIZARD_REMARK_EVENT_NAME, { remark });
Two problems with this:
1. It's not sanitized (security)
The remark is untrusted, model-generated text captured from output and sent to PostHog analytics with no sanitization. It can carry things we don't want to ingest verbatim: secrets/API keys the model echoed from the environment, file paths or PII, prompt-injection payloads, or control/escape characters. We should sanitize before capturing — strip control chars, redact secret-shaped tokens, cap length, and validate it looks like a single reflection line rather than arbitrary content.
2. Parsing it out of prose is flaky
Marker-scanning the agent's output stream is fragile. We've already hit failure modes:
- The model echoes the ask itself instead of a real remark (fixed narrowly in #806 / the
REMARK_INSTRUCTIONecho check inremark()). - Only the first line after the marker is captured; multi-line or marker-mid-line output is dropped or truncated.
- Any change to prose formatting can silently break extraction.
As noted in the thread: parsing wizard logs/output for remarks is inherently flaky and should be reworked.
Proposed direction
Move remarks off free-text prose parsing onto a structured channel — e.g. have the agent emit the remark via a dedicated tool call / structured output field instead of an inline marker, so it arrives as a typed value we don't have to regex out of the stream. Then run whatever we receive through a sanitization pass before analytics.capture.
Acceptance criteria
- Remark content is sanitized (control chars stripped, secret-shaped tokens redacted, length-capped) before being captured to analytics.
- Remark delivery no longer depends on regex-scanning prose — it comes through a structured/typed channel.
- Echo-of-the-ask and empty/garbage remarks are reliably rejected.
- Tests cover sanitization and the new extraction path.
Context: found while debugging the Windows skill-install issue; remarks broke (#806) and Vincent flagged the parsing approach as flaky and wanting a rework.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/lib/agent/output-signals.ts, including AgentOutputSignals.remark() and the REMARK_INSTRUCTION echo check, then trace the analytics.capture call for WIZARD_REMARK_EVENT_NAME. Review the existing remark flow and determine where the structured delivery and sanitization should be tested; done means typed remark delivery, rejected echoes and empty values, and sanitized analytics payloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- security, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100