dwmkerr / dwmkerr/shellwright

feat: recording evaluation workflow for PRs

Open
#41 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
37
Forks
5
Avg merge
3d 23h
Merged PRs (30d)
1

Description

## Summary

Create a CI workflow that generates test recordings on every PR, allowing visual verification of recording quality. This is foundational infrastructure that other features depend on.

## Motivation

Features like adaptive frame timing (#37 typewriter, future frame deduplication) need visual verification. Automated tests can check file sizes and frame counts, but humans need to see the actual recordings to verify quality.

## Proposed Design

### Evaluation Scripts

```
evaluations/
├── scripts/
│ ├── typewriter-demo.ts # Keystroke-by-keystroke text entry
│ ├── idle-heavy.ts # Long pauses between actions
│ ├── rapid-output.ts # Fast-scrolling output (e.g., npm install)
│ └── vim-session.ts # Interactive editor session
├── baseline/ # Reference recordings (committed)
└── README.md # Documents each test case
```

Each script uses shellwright MCP tools to generate a deterministic recording.

### CI Workflow

New workflow: `.github/workflows/recording-eval.yaml`

```yaml
name: Recording Evaluation
on: pull_request

jobs:
generate-recordings:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci && npm run build

- name: Generate recordings
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: npm run eval:generate

- name: Generate comparison table
run: npm run eval:compare >> $GITHUB_STEP_SUMMARY

- uses: actions/upload-artifact@v4
with:
name: recordings
path: evaluations/generated/
```

### PR Comment with Comparison Table

Auto-generated comment on each PR:

| Test Case | Baseline | Current | Frames | Size | Status |
|-----------|----------|---------|--------|------|--------|
| typewriter | ![base](url) | ![current](url) | 45 → 42 | 340KB → 320KB | ✅ |
| idle-heavy | ![base](url) | ![current](url) | 120 → 125 | 890KB → 910KB | ⚠️ +2% |

### Requirements

- **AI credentials**: `ANTHROPIC_API_KEY` secret for running recordings
- **Deterministic scripts**: Same prompts should produce similar (not identical) recordings
- **Baseline management**: Way to update baselines when intentional changes are made

## Open Questions

- [ ] Should this block merge or just be informational?
- [ ] How to handle baseline updates (manual commit vs automated)?
- [ ] Threshold for size regression warnings?

## Blocked By

None - this is foundational.

## Blocks

- Adaptive frame timing (future issue)
- Typewriter feature (#37)
- Any future recording quality changes

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.