BOHICA-LABS / BOHICA-LABS/vsdd-factory
docs(observability): document WHEN to enable the observability stack within the pipeline lifecycle (currently undocumented)
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
vsdd-factory ships three layers of observability:
1. **File-based event emission** — always on by default; logs to `.factory/logs/events-YYYY-MM-DD.jsonl`.
2. **Docker stack** — opt-in; OTel Collector + Loki + Prometheus + Grafana + image renderer; 7 auto-provisioned dashboards. Enabled via `factory-obs register` + `factory-obs up`.
3. **Claude Code OTel telemetry** — opt-in; 5 env vars in `.claude/settings.local.json`. Pipes Claude Code's own metrics (cost, tokens, sessions) into the same stack.
Each layer has its own enable path (`/vsdd-factory:onboard-observability`, `factory-obs up`, `/vsdd-factory:claude-telemetry on`). But **no document covers when in the pipeline lifecycle an operator should enable layers 2 and 3.**
## What's missing
The following questions have no documented answers:
1. **Should I enable observability BEFORE `/factory-health`, or AFTER?**
- Today: ambiguous. `/onboard-observability` says "Brand new project that just had the vsdd-factory plugin installed." `/factory-health` makes no mention of observability ordering. (Filed as issue #203 — these two skills race when run in the wrong order.)
2. **Is observability required for pipeline phases to run?**
- Today: not documented. The pipeline appears to work without it — file-based emission to `.factory/logs/` happens regardless. The Docker stack is purely a query/dashboard surface.
3. **At what point in greenfield / brownfield / feature mode should I expect to start the stack?**
- Today: not documented. Greenfield/brownfield/feature orchestrator sequences (`agents/orchestrator/*-sequence.md`) make NO mention of observability — `grep -r 'observab\|factory-obs\|telemetry'` returns zero hits in any orchestrator sequence file.
4. **What happens if I never enable the Docker stack?**
- Today: implicit. The file-based JSONL is still present; you can query with jq or `factory-replay`. The stack adds Grafana dashboards but doesn't change pipeline behavior. The docs don't make this explicit.
5. **Should observability be on for short experiments (e.g., trying out the factory on one repo)?**
- Today: no guidance. The file-based layer is always on; the Docker stack is heavyweight (5 containers, ~30 days retention). For a 1-hour experiment, the stack is overkill but the docs don't say so.
## Predicted correct ordering (my best read of the evidence)
```
1. Install plugin
2. /vsdd-factory:factory-health ← bootstrap .factory/ worktree
3. /vsdd-factory:onboard-observability ← register + telemetry config
4. factory-obs up ← start the stack (if you want dashboards)
5. Restart Claude Code session ← OTel env vars take effect
6. Begin pipeline (greenfield/brownfield/feature)
```
But this is reverse-engineered, not documented.
## Proposed
### 1. Add a "Lifecycle position" section to `docs/guide/observability.md`
```markdown
## When to enable observability
| Layer | When | Action |
|-------|------|--------|
| File-based emission | Automatic — active from plugin install | (none — `VSDD_TELEMETRY=off` to disable) |
| Docker stack | After `/factory-health` bootstraps `.factory/`, before pipeline phases run | `/vsdd-factory:onboard-observability` then `factory-obs up` |
| Claude Code OTel | Same as Docker stack | bundled in onboard-observability |
### Recommended ordering
For a fresh project: `factory-health` → `onboard-observability` → `factory-obs up` → restart session → begin pipeline.
For an existing factory-equipped project: just `factory-obs up` if the stack isn't running and you registered earlier.
For a 1-hour experiment / read-only spike: skip the Docker stack entirely; rely on file-based emission + `factory-replay` CLI.
```
### 2. Reference observability ordering in `agents/orchestrator/*-sequence.md`
Add a step S0 or pre-flight to each orchestrator sequence:
```markdown
S0. Observability check (operator confirms before starting)
- Is the Docker stack running? `factory-obs status`
- If you want dashboards during this pipeline run, start it before S1.
- The pipeline does not depend on the stack; file-based emission is always on.
```
### 3. Cross-reference in `/vsdd-factory:onboard-observability` SKILL.md
Add an explicit "Lifecycle position" section:
```markdown
## Lifecycle position
This skill should run:
- AFTER `/vsdd-factory:factory-health` (which bootstraps the `.factory/` worktree).
Running this skill first creates `.factory/logs/` in a plain directory, which
blocks factory-health's worktree mount. (See issue #203 for the bug.)
- BEFORE any orchestrator pipeline phase (greenfield/brownfield/feature), so you
have observability data from the first agent dispatch onward.
```
### 4. README pipeline flow diagram
Add observability as a sidecar lane in the README's pipeline mermaid diagram, so operators see it visually positioned.
## Acceptance criteria
- [ ] `docs/guide/observability.md` has a "When to enable" / "Lifecycle position" section.
- [ ] Each orchestrator sequence (`*-sequence.md`) has a pre-flight or S0 step that mentions observability ordering.
- [ ] `/vsdd-factory:onboard-observability` SKILL.md explicitly states ordering relative to `/vsdd-factory:factory-health`.
- [ ] README mentions observability layer/lifecycle in the quickstart.
- [ ] No operator should have to reverse-engineer the ordering from the source.
## Found during
Reviewing observability docs on `switchboard-blue` (2026-06-23, vsdd-factory@1.0.0-rc.21). After running `/onboard-observability` first (which is what the skill's "Brand new project" guidance implies), then `/factory-health`, I hit the race documented in #203. The correct ordering only became clear by reading the SKILL.md files directly — not from any user-facing doc.
## Related
- (filed this session) #203 — onboarding ordering bug; this issue is the DOCS counterpart that would have prevented the bug from being hit at all
- (pending this session) P3, P4 — preflight gaps; if those existed, a single preflight would document the order implicitly by running everything in the right sequence
Contributor guide
Assessment
This issue has not been assessed yet.