BOHICA-LABS / BOHICA-LABS/vsdd-factory

docs(observability): document WHEN to enable the observability stack within the pipeline lifecycle (currently undocumented)

Open
#237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

vsdd-factory ships three layers of observability:

1. **File-based event emission** — always on by default; logs to `.factory/logs/events-YYYY-MM-DD.jsonl`.
2. **Docker stack** — opt-in; OTel Collector + Loki + Prometheus + Grafana + image renderer; 7 auto-provisioned dashboards. Enabled via `factory-obs register` + `factory-obs up`.
3. **Claude Code OTel telemetry** — opt-in; 5 env vars in `.claude/settings.local.json`. Pipes Claude Code's own metrics (cost, tokens, sessions) into the same stack.

Each layer has its own enable path (`/vsdd-factory:onboard-observability`, `factory-obs up`, `/vsdd-factory:claude-telemetry on`). But **no document covers when in the pipeline lifecycle an operator should enable layers 2 and 3.**

## What's missing

The following questions have no documented answers:

1. **Should I enable observability BEFORE `/factory-health`, or AFTER?**
- Today: ambiguous. `/onboard-observability` says "Brand new project that just had the vsdd-factory plugin installed." `/factory-health` makes no mention of observability ordering. (Filed as issue #203 — these two skills race when run in the wrong order.)

2. **Is observability required for pipeline phases to run?**
- Today: not documented. The pipeline appears to work without it — file-based emission to `.factory/logs/` happens regardless. The Docker stack is purely a query/dashboard surface.

3. **At what point in greenfield / brownfield / feature mode should I expect to start the stack?**
- Today: not documented. Greenfield/brownfield/feature orchestrator sequences (`agents/orchestrator/*-sequence.md`) make NO mention of observability — `grep -r 'observab\|factory-obs\|telemetry'` returns zero hits in any orchestrator sequence file.

4. **What happens if I never enable the Docker stack?**
- Today: implicit. The file-based JSONL is still present; you can query with jq or `factory-replay`. The stack adds Grafana dashboards but doesn't change pipeline behavior. The docs don't make this explicit.

5. **Should observability be on for short experiments (e.g., trying out the factory on one repo)?**
- Today: no guidance. The file-based layer is always on; the Docker stack is heavyweight (5 containers, ~30 days retention). For a 1-hour experiment, the stack is overkill but the docs don't say so.

## Predicted correct ordering (my best read of the evidence)

```
1. Install plugin
2. /vsdd-factory:factory-health ← bootstrap .factory/ worktree
3. /vsdd-factory:onboard-observability ← register + telemetry config
4. factory-obs up ← start the stack (if you want dashboards)
5. Restart Claude Code session ← OTel env vars take effect
6. Begin pipeline (greenfield/brownfield/feature)
```

But this is reverse-engineered, not documented.

## Proposed

### 1. Add a "Lifecycle position" section to `docs/guide/observability.md`

```markdown
## When to enable observability

| Layer | When | Action |
|-------|------|--------|
| File-based emission | Automatic — active from plugin install | (none — `VSDD_TELEMETRY=off` to disable) |
| Docker stack | After `/factory-health` bootstraps `.factory/`, before pipeline phases run | `/vsdd-factory:onboard-observability` then `factory-obs up` |
| Claude Code OTel | Same as Docker stack | bundled in onboard-observability |

### Recommended ordering

For a fresh project: `factory-health` → `onboard-observability` → `factory-obs up` → restart session → begin pipeline.

For an existing factory-equipped project: just `factory-obs up` if the stack isn't running and you registered earlier.

For a 1-hour experiment / read-only spike: skip the Docker stack entirely; rely on file-based emission + `factory-replay` CLI.
```

### 2. Reference observability ordering in `agents/orchestrator/*-sequence.md`

Add a step S0 or pre-flight to each orchestrator sequence:

```markdown
S0. Observability check (operator confirms before starting)
- Is the Docker stack running? `factory-obs status`
- If you want dashboards during this pipeline run, start it before S1.
- The pipeline does not depend on the stack; file-based emission is always on.
```

### 3. Cross-reference in `/vsdd-factory:onboard-observability` SKILL.md

Add an explicit "Lifecycle position" section:

```markdown
## Lifecycle position

This skill should run:
- AFTER `/vsdd-factory:factory-health` (which bootstraps the `.factory/` worktree).
Running this skill first creates `.factory/logs/` in a plain directory, which
blocks factory-health's worktree mount. (See issue #203 for the bug.)
- BEFORE any orchestrator pipeline phase (greenfield/brownfield/feature), so you
have observability data from the first agent dispatch onward.
```

### 4. README pipeline flow diagram

Add observability as a sidecar lane in the README's pipeline mermaid diagram, so operators see it visually positioned.

## Acceptance criteria

- [ ] `docs/guide/observability.md` has a "When to enable" / "Lifecycle position" section.
- [ ] Each orchestrator sequence (`*-sequence.md`) has a pre-flight or S0 step that mentions observability ordering.
- [ ] `/vsdd-factory:onboard-observability` SKILL.md explicitly states ordering relative to `/vsdd-factory:factory-health`.
- [ ] README mentions observability layer/lifecycle in the quickstart.
- [ ] No operator should have to reverse-engineer the ordering from the source.

## Found during

Reviewing observability docs on `switchboard-blue` (2026-06-23, vsdd-factory@1.0.0-rc.21). After running `/onboard-observability` first (which is what the skill's "Brand new project" guidance implies), then `/factory-health`, I hit the race documented in #203. The correct ordering only became clear by reading the SKILL.md files directly — not from any user-facing doc.

## Related

- (filed this session) #203 — onboarding ordering bug; this issue is the DOCS counterpart that would have prevented the bug from being hit at all
- (pending this session) P3, P4 — preflight gaps; if those existed, a single preflight would document the order implicitly by running everything in the right sequence

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.