BOHICA-LABS / BOHICA-LABS/vsdd-factory

feat(observability): OTEL metrics have no project/instance dimension — cost & velocity can't be attributed per pilot or per parallel run

Open
#324 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Factory OTEL metrics carry **no project / cwd / repo / instance dimension**, so Grafana cannot attribute cost, tokens, commits, or active-time to a specific factory pilot — or distinguish parallel runs of the same pilot. With multiple factory instances on one machine (e.g. `ftc-blue`, `switchboard-blue`, `akey`), every `claude_code_*` series collapses into one undifferentiated aggregate. This makes per-project velocity, cost, and any **compaction/context-spiral** analysis structurally impossible from the obs stack.

## What exists today

Every `claude_code_*` metric in Prometheus is labelled with `session_id` + machine/model attributes, but **nothing identifies the project**. Verified across all five emitted metrics:

| Metric | project/cwd/repo/instance label? |
|---|---|
| `claude_code_token_usage_tokens_total` | **none** |
| `claude_code_cost_usage_USD_total` | **none** |
| `claude_code_commit_count_total` | **none** |
| `claude_code_session_count_total` | **none** |
| `claude_code_active_time_seconds_total` | **none** |

The only disambiguator is `session_id` — an opaque UUID (9 distinct over the last 7 days). There is no `project`, `cwd`, `repo`, `workspace`, `branch`, `worktree`, or `instance` label on any series.

## Why this matters

1. **No per-project view.** Cost/tokens/commits/active-time for three concurrently-running pilots sum into one series. A per-project cost regression or velocity decline is invisible by construction.
2. **No parallel-instance view.** On 2026-06-28 `ftc-blue` ran 4 distinct sessions, `switchboard-blue` 2, `akey` 2. Even *with* a `project` label, two concurrent runs of one project would still collapse together — there is no `instance`/`run_id`/`worktree` dimension to separate them.
3. **Blocks death-spiral / compaction analysis.** Detecting shrinking inter-compaction intervals, rising context-fill, or falling per-project velocity all require slicing the existing counters per project/instance first. (Related: the compaction-observability cluster #317/#318/#319/#320 — all of it is unsliceable per-project until this label exists.)

## The data exists — it just isn't emitted as a label

The factory's own `dispatcher-internal-*.jsonl` records `session_id` next to the project (by log location), so the mapping is recoverable by an **offline join**: e.g. `switchboard-blue`'s dispatcher session `4016dfcc-…` is the same UUID present on `claude_code_cost_usage_USD_total{session_id=~"4016dfcc.*"}` in Prometheus. This proves the attribution is *knowable at session start* — it's simply not attached to the metrics.

The offline join is not a substitute: it breaks when logs rotate/prune, needs a manual two-source merge, and still cannot separate parallel instances.

## Proposed change

Emit resource attributes known at session start so they land as Prometheus labels (via `OTEL_RESOURCE_ATTRIBUTES`, or the session-start telemetry hook):

- `project` — product/repo name (from `.factory/STATE.md` `product:` or git toplevel)
- `cwd` — absolute working directory
- `instance` / `worktree` — to separate concurrent runs of the same project
- (nice-to-have) `cycle` / `wave` — for in-pipeline correlation

Then Grafana gets per-project and per-instance breakdowns — the precondition for any velocity-drift, cost-attribution, or compaction-cadence panel.

## Constraints (SOUL.md alignment)

- **No phone-home.** Labels are derived locally from STATE.md / git / cwd and shipped to the user's own collector, same as every existing metric.
- **Gradual elaboration.** `project` + `cwd` alone unblocks most of the value; `instance`/`cycle` can follow when a concrete parallel-run analysis needs them.

## Environment

- vsdd-factory develop tip; `claude-telemetry` + `factory-obs` (Grafana 13.0.1, Prometheus :9090, Loki :3100).
- Observed across three live pilots: `~/work/ftc-blue`, `~/work/switchboard-blue`, `~/work/aae-orc/akey`.
- Metric scope: `com.anthropic.claude_code` v2.1.195 (`service_name=claude-code`).

---

🤖 Filed by **beadle** running as `arcavenai`, an automated triage assistant. Surfaced while investigating a per-project compaction-cadence question against the live obs stack; evidence verified directly against Prometheus and the dispatcher logs. Detail is summarized, not replicated — follow the cross-referenced issues to fetch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.