jlevy / jlevy/tryscript

Preserve named patterns when --update rewrites a failing block

Open
#49 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
13
Forks
1
Avg merge
3h 38m
Merged PRs (30d)
3

Description

## Summary

`tryscript run --update` replaces the entire expected-output section of each failing block with literal output. If that block contains named patterns for values such as a sandbox path, timestamp, PID, or allocated byte count, the update silently turns them into one run's concrete values. The next ordinary run then fails even when the intended stable-output update was correct.

This makes the documented update-and-compare workflow unsafe for realistic golden files that intentionally retain named patterns.

Observed with tryscript 0.1.7. The current `buildUpdatedBlock()` implementation writes `result.actualOutput` directly, so this appears to be current behavior rather than a platform-specific failure.

## Minimal Reproduction

```markdown
---
patterns:
PID: '\d+'
---

# Mixed Stable and Unstable Output

```console
$ node -e "console.log('status: new'); console.log('pid: ' + process.pid)"
status: old
pid: [PID]
? 0
```
```

Run:

```console
tryscript run --update example.tryscript.md
tryscript run example.tryscript.md
```

The first command rewrites the block to a literal PID. The second command fails when Node has a different PID.

A concrete downstream case changed fixture names and stable byte totals in CLI transcripts containing `[SCAN_PATH]`, `[MTIME_NS]`, and `[ALLOCATED]`. `--update` replaced all three pattern classes with literals across every changed block, and the mandatory comparison failed on the next sandbox.

## Desired Behavior

When updating a structurally corresponding block:

- update stable text that no longer matches
- preserve named patterns where the old expected output used them and the corresponding new value still matches that pattern
- make an immediate ordinary rerun pass
- warn clearly instead of silently literalizing a named pattern when correspondence is ambiguous
- retain an explicit opt-in such as `--expand-all` for users who do want named patterns replaced by concrete values

The active wildcard-expansion plan already notes possible integration where update mode prefers preservation when possible. A public issue would make that remaining workflow gap visible and testable.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the current buildUpdatedBlock() implementation and review the active wildcard-expansion plan for update-mode integration. Reproduce the failure with the provided markdown example, then trace how expected output and named patterns correspond. Done means stable text updates while matching named patterns are preserved, ambiguous cases warn, and an ordinary rerun passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.