afx-team / afx-team/hebb-mind

ci: assert every hebb command in plugin.json / hooks.json / docs resolves to a real Click command

Aperta
#30 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area: usability effort: simple enhancement
Lingua principale
Python
Stelle
52
Fork
18
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Context

Goal axis: **好用 / usability** — permanently close the "advertised command does not exist" regression loop that both audits kept rediscovering.

The original failure (audit `core-system-audit-2026-06-07.md`, finding C0-2) was that `.claude-plugin/plugin.json` and `.codex/hooks.json` invoked a command that did not exist: the group is registered as `@click.group("claude-code")` (so `hebb cc ...` resolves to "No such command"), and there was never a `write` subcommand (UserPromptSubmit should call `prompt`). Meanwhile the programmatic installer wrote the *correct* strings, so two install paths silently drifted apart — breaking recall + memory for anyone who installed via the plugin marketplace or that shipped `hooks.json`. The newuser audit (`newuser-experience-audit-2026-06-08.md`, finding U3) found the same `hebb claude-code write` ghost command duplicated across the docs (`cli.md`, `claude-code.md`, README).

PR #24 fixed the strings, but a string fix does not prevent the *next* drift. This issue is about the durable guard.

## Current state (verified against the tree)

A guard for the **manifest + Dockerfile** surface already landed in PR #24 (commit `9244265`) — so this issue is **narrowed** to the remaining gap (docs / examples / READMEs), not the whole thing.

What already exists and passes (`pytest tests/unit/test_audit_distribution.py` → 3 passed):

- `tests/unit/test_audit_distribution.py:97` `test_plugin_hook_commands_resolve` — extracts every hook `command` from `.claude-plugin/plugin.json` and resolves each against the live Click tree.
- `tests/unit/test_audit_distribution.py:107` `test_codex_hook_commands_resolve` — same for `.codex/hooks.json`.
- `tests/unit/test_audit_distribution.py:117` `test_dockerfile_cmd_references_real_serve_command` — same for the `docker/Dockerfile` `CMD` chain.
- `tests/unit/test_audit_distribution.py:26` `_resolve_command(...)` — the shared Click-tree walker (`from hebb.cli.main import main`); reused below.
- This file is already in CI: `.github/workflows/ci.yml:51-55` runs `pytest tests/unit ...` on every OS/Python matrix cell.

Grounding of the shipped strings (all correct today, confirming PR #24 landed):

- `.claude-plugin/plugin.json:20,32,44` → `hebb claude-code recall` / `prompt` / `stop`.
- `.codex/hooks.json:9,18,27` → `hebb claude-code recall` / `prompt` / `stop`.
- `src/hebb/integrations/claude_code/cli.py:8` → `@click.group("claude-code")`; subcommands at lines `install` (20), `uninstall` (34), `recall` (42), `prompt` (50), `stop` (58). Live introspection confirms `claude-code` subs = `install, prompt, recall, stop, uninstall`.

**The remaining gap** — no test resolves `hebb ...` strings embedded in human-facing docs, which is exactly where the `write` ghost command lived (U3):

- No test under `tests/` references `repo_pages`, `examples`, or `README` (grep is empty).
- `repo_pages/**/*.md` contains ~109 `hebb ` strings (e.g. `repo_pages/guide/claude-code.md:25,42,43,49`), with an EN tree and a `repo_pages/zh/` mirror.
- `examples/03_mcp_quickstart.md` and `examples/README.md` contain `hebb ...` command strings (e.g. `examples/03_mcp_quickstart.md:22,25,33`).
- `README.md` and `README_ZH.md` both carry `hebb ...` invocations.

So a future doc typo (`hebb claude-code wrtie`, a renamed/removed subcommand, etc.) would ship green today.

## Proposed approach

Extend the existing `tests/unit/test_audit_distribution.py` (reuse its `_resolve_command` walker and `from hebb.cli.main import main` introspection — do not duplicate the tree logic) with a docs/examples/README coverage test:

1. Walk `README.md`, `README_ZH.md`, `examples/**/*.md`, and `repo_pages/**/*.md` (skip `repo_pages/node_modules/`).
2. Extract every `hebb ...` invocation from both fenced code blocks (```` ```bash ````/```` ```sh ````/plain) and inline `` `hebb ...` `` spans.
3. Normalize each invocation to a token path (strip the `hebb` executable; stop the sub-command path at the first `-`/`--` option or shell metachar, matching the existing `_hebb_invocations` helper).
4. Assert each resolves to a registered command + subcommand via `_resolve_command`. Allow a small, explicit allowlist for intentionally illustrative/placeholder strings (e.g. `hebb ...` literally, or `hebb ` help syntax) so the test stays low-noise.
5. The test runs in the existing CI `test` job automatically (it lives under `tests/unit`).

Keep the manifest/Dockerfile tests as-is; this only adds the docs/examples/README dimension.

## Acceptance criteria

- [ ] A test introspects the live Click command tree from the `hebb` entrypoint (reuses the existing `_resolve_command` / `main` import, no duplicate tree logic).
- [ ] It extracts and resolves `hebb ...` invocations from `README.md`, `README_ZH.md`, `examples/**/*.md`, and `repo_pages/**/*.md` (EN + `repo_pages/zh/` mirror), covering both fenced blocks and inline code spans.
- [ ] Introducing a fabricated bad command (e.g. `hebb claude-code wrtie` or `hebb foo bar`) in any covered doc fails the test; the current tree passes.
- [ ] The manifest coverage (`.claude-plugin/plugin.json`, `.codex/hooks.json`) and Dockerfile coverage remain green (no regression to the existing three tests).
- [ ] The new test runs in CI on every PR (verified via `.github/workflows/ci.yml` `test` job picking up `tests/unit`).
- [ ] Any intentional placeholder strings are handled via a narrow, documented allowlist rather than broad regex loosening.

## Scope / out of scope

- In scope: a CI-enforced assertion that every `hebb ...` string advertised in shipped manifests **and** human-facing docs/examples/READMEs resolves to a real Click command+subcommand.
- Out of scope: validating option flags/argument values (e.g. that `--scope user` is a legal choice) — only the command-path is resolved, matching the existing `_resolve_command` semantics. Out of scope: localized prose translation correctness, and non-`hebb` shell commands in fences.
- Note: the manifest + Dockerfile half already shipped in PR #24; this issue is the doc/example/README extension only.

This is the highest-leverage usability fix in the backlog because it permanently closes the doc/command-drift regression loop that the `core-system-audit-2026-06-07` (C0-2) and `newuser-experience-audit-2026-06-08` (U3) audits independently rediscovered.

## References

- `reports/audit/core-system-audit-2026-06-07.md` (finding C0-2 — `hebb cc write` / wrong group name)
- `reports/audit/newuser-experience-audit-2026-06-08.md` (finding U3 — `hebb claude-code write` doc drift)
- `reports/design/capability-gap-roadmap-2026-06-11.md`
- `tests/unit/test_audit_distribution.py` (existing manifest/Dockerfile guard to extend)

Filed from the capability-gap roadmap (reports/design/capability-gap-roadmap-2026-06-11.md).

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.