New skill: /qa-headless — QA for backend features with no UI (cron jobs, workers, webhooks, notifiers, CLIs)
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
## Summary
gstack's QA surface is entirely browser-based — `/qa`, `/qa-only`, `/browse`, `/benchmark`, and `/canary` all drive a headless browser against a running web app. That's great for frontends, but a huge class of backend features has no UI at all and is effectively untestable with current gstack tooling. Propose a new `/qa-headless` skill that complements `/qa` and covers the backend gap.
## Problem
Backend features with no UI have no gstack QA tool today:
- Scheduled jobs (cron, Heroku Scheduler, APScheduler, Celery beat)
- Queue workers (Celery, Sidekiq, BullMQ, Faktory)
- Webhook handlers whose observable output is a side-effect, not an HTTP response
- Notification senders (Slack, email, SMS, push)
- CLI tools and management commands
- Data pipelines / ETL jobs
- Backend services whose "output" is a row written, a message sent, or a file produced
For these, `/qa` is useless. The real QA loop looks like: boot or isolate → trigger the thing → capture the side-effect (Slack Block Kit JSON, MIME email, DB row, log line) → eyeball or diff against expected. No skill does this today.
## Prior art
Issue #608 (closed, no resolution) raised the same gap — a user ran `/qa` on a backend and found results unreliable. No existing skill or PR addresses the gap.
## Proposed solution: `/qa-headless`
Core behaviors:
1. **Detect feature shape.** From git diff + framework conventions (Procfile, Celery tasks, FastAPI routes, management commands, Rails jobs, Go cmd/, Node scripts, etc.), decide whether the changed code is a cron/job, webhook handler, CLI, queue worker, or notifier.
2. **Find or propose a dry-run harness.** Use an existing `--dry-run` flag or test-mode env var if present. If not, suggest adding one as part of the PR (integrates with `/review`).
3. **Capture side-effects structurally.** Intercept outgoing HTTP (Slack webhooks, Twilio, email APIs) via monkeypatch or local proxy, then render the captured payload human-readably — Slack Block Kit → faux-Slack preview, MIME → subject+body, API calls → parameter table.
4. **Diff against expected.** Optional golden-file diff for regression detection.
5. **Interactive fix loop.** Find issue → propose fix → apply → re-run dry-run → confirm output changed → commit atomically. Same shape as `/qa`.
6. **Produce a report.** Pass/fail per scenario, integrates with `/ship` review readiness dashboard.
## Out of scope
- Don't duplicate `/qa` (UI features). If the feature has a UI, route to `/qa`.
- Don't duplicate `/review` (static code review).
- Don't duplicate `/investigate` (post-hoc debugging).
- Framework- and language-agnostic: detect Python (FastAPI/Flask/Django), Node (Express/Fastify), Ruby (Rails), Go, and others via convention. A skill is prompt logic, not a runtime — no reason to restrict stacks.
- Prefer isolation over full-app-boot where possible — faster, more reproducible.
## Integrations with existing skills
- `/plan-eng-review` and `/office-hours` should recommend `/qa-headless` during test planning when the feature has no UI.
- `/review` should flag PRs that add a cron job or notifier without a `--dry-run` flag as a testing gap.
- `/ship` should include `/qa-headless` in the review readiness dashboard (same tier as `/qa` — optional, recommended when applicable).
- `/health` stays separate — it runs pytest/jest (code compiles + unit tests); `/qa-headless` is "did the side-effect look right."
## Motivating test case
Daily Slack call digest cron at 10am CT. Backend cron job. Groups `CallSession` rows by `(caller_e164, device_id)`, POSTs one Block Kit message per group to a Slack webhook. No UI.
`/qa-headless` should help:
1. Run `scripts/run_call_digest.py --date=2026-04-15 --dry-run` and see the Slack output pretty-printed in the terminal.
2. Optionally POST to a `--channel=test` webhook and open the Slack thread.
3. Report: "9 groups, 47 calls, 1 unrouted subsection, Block Kit valid, ship-ready" — or "0 groups found — date boundary bug?"
If the skill handles that case end-to-end, it's ready.
## Acceptance criteria
- [ ] `/qa-headless` skill lives in the gstack skills directory with a SKILL.md following gstack conventions (preamble, voice, completion status, telemetry, learnings).
- [ ] Skill detects feature shape (cron/job, webhook, CLI, queue worker, notifier) from git diff + framework conventions across Python, Node, Ruby, and Go projects.
- [ ] Skill uses an existing `--dry-run` harness if present; otherwise proposes adding one.
- [ ] Skill captures outgoing HTTP to Slack, Twilio, and email APIs and renders payloads human-readably (Block Kit preview, MIME subject+body, API parameter table).
- [ ] Interactive fix loop: find → fix → re-run → commit atomically.
- [ ] Produces a pass/fail report compatible with `/ship` review readiness.
- [ ] Main gstack README includes a short blurb positioning `/qa-headless` relative to `/qa`.
- [ ] Motivating test case (daily Slack call digest cron) runs end-to-end and produces the expected report.
## QA instructions
1. In a Python/FastAPI repo with an existing backend cron script that POSTs to Slack, invoke `/qa-headless` — skill detects the feature as a cron job. Expected: correct shape classification printed.
2. Repeat in a Node/Express repo with a BullMQ worker — skill detects the feature as a queue worker. Expected: correct shape classification printed.
3. Repeat in a Rails repo with an ActiveJob notifier — skill detects the feature as a notifier. Expected: correct shape classification printed.
4. Run the skill against a script that already has `--dry-run` — skill invokes it without proposing to add the flag. Expected: dry-run executed, Slack Block Kit payload rendered as faux-Slack preview in terminal.
5. Run the skill against a script without `--dry-run` — skill proposes adding one and offers to apply the change. Expected: proposal shown with diff; nothing committed without approval.
6. Run the motivating test case (`scripts/run_call_digest.py --date=2026-04-15 --dry-run`). Expected: summary report matches the form "N groups, M calls, Block Kit valid, ship-ready".
7. Introduce a bug in the cron script (e.g., wrong date boundary) and re-run — skill reports the anomaly ("0 groups found") rather than silently passing. Expected: clear failure signal.
8. Invoke `/qa` on a backend-only feature — existing skill routes the user to `/qa-headless` instead. Expected: routing message, no browser launched.
## Context
Filed by a gstack daily user (technical founder, Python/FastAPI/Postgres on Heroku, healthcare telephony) who hit this gap today while shipping a daily Slack digest cron. Wants a tight, no-fluff, interactive-but-fast, framework-aware skill. Related: #608.
## Notes
- Name bikeshed is settled: `/qa-headless` chosen to pair cleanly with `/qa` (browser-driven) and signal "happens where you can't see it."
- Language/framework support is not artificially scoped — a skill is prompt logic, so Python, Node, Ruby, Go, and others should all be in from day one.
- Integration edits to `/plan-eng-review`, `/office-hours`, `/review`, and `/ship` are proposed but should be tracked as follow-up issues, not bundled into this one.
Contributor guide
Assessment
This issue has not been assessed yet.