Comfy-Org / Comfy-Org/ComfyUI_frontend

[test-recorder] Secret scrubbing misses multiline/unlabeled input; raw autosave written to disk unscrubbed

Open
#15,839 0 comments 0 reactions 1 assignee Claimed by @christian-byrne View on GitHub
area:testing Bug Potential Bug
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 7h
Merged PRs (30d)
490

Description

### Problem
The secret scrubber in `tools/test-recorder` has gaps, and raw (unscrubbed) recorded code is written to disk during recording:

1. `src/transform/scrub.ts:6-10,17-20,45-48` — scrubbing is line-based: each line must independently contain both a sensitive label and a typing call. A password recorded as a multiline locator/action chain, or typed into a field identified only by test ID, survives scrubbing. Short values also evade the long-token heuristic.
2. `src/recorder/template.ts:57-85,145-154` + `src/recorder/runner.ts:172-180` — during recording, Playwright continuously autosaves raw generated code (including any typed credentials) to `browser_tests/tests/_recorded-code.txt` in plaintext. Sanitization only happens after recording completes (`src/transform/engine.ts:24-25`).
3. Detection doesn't cover common API-key formats (e.g. `sk-`, `ghp_`, `AKIA` prefixes).

This already happened in live user testing: a typed cloud password landed in generated output.

### Suggested fix
- Make scrubbing structure-aware (associate locator chains with their fill/type actions across lines; conservatively redact typing into password-type inputs and credential-labelled locators).
- Scrub the autosave stream continuously, or write it to a private temp location with restrictive permissions.
- Add API-key-format patterns and an always-visible pre-record reminder that credentials should only be entered before pressing Record.

_Line references are against branch `cb/comfy-test-usability` (PR #15837); adjust after merge._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.