/plan-tune inferred psychographic never leaves 0.5: SIGNAL_MAP is keyed by stable option keys, the capture hook logs UI labels (0 of 1,286 rows match)
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
## Summary
`/plan-tune`'s inferred psychographic never leaves its 0.5 seed, on a log with
1,286 events where all four calibration gates pass. The derivation code is
fine — I proved it works by hand. Nothing in normal operation can reach it.
Measured on v1.67.1.0 (`c86e647`), against a real 20-day, 1,286-row log.
## The machinery works. It is starved.
I logged nine `AskUserQuestion` answers using **registered ids with stable
option keys** instead of the ad-hoc ids the skills actually emit. That satisfies
every gate at once, and the pipeline responded immediately:
```
before: architecture_care 0.5 (0 contributing events)
after : architecture_care 0.673 (5 contributing events)
$ gstack-developer-profile --trace architecture_care
TRACE: 5 events for architecture_care
plan-eng-review-arch-finding -> fix-now (+0.05) x4
plan-eng-review-test-gap -> fix-now (+0.04)
```
So this is a coverage problem, not a repair problem. That is the whole issue in
one command, and it is cheap to reproduce.
## Why no real question reaches it
`--derive` needs three exact-string matches to all land
(`bin/gstack-developer-profile:355`, `:356`, `:423`):
| Gate | Rows passing (of 1,286) |
|---|---|
| 1. `QUESTIONS[question_id]` | **5** (3 distinct ids of 983) |
| 2. that entry has a `signal_key` | **3** |
| 3. `SIGNAL_MAP[signal_key][user_choice]` | **0** |
**Gate 3 is the categorical one, and it is the one registry growth cannot fix.**
`QuestionDef.options` is typed `StandardOption[]` — stable keys like `accept`,
`expand`, `fix-now` — and `SIGNAL_MAP` is keyed by those. But the capture hook
records the **UI label**:
```json
{"question_id":"plan-eng-review-ci-typecheck","user_choice":"Restore the parallel typecheck job"}
```
Claude Code's `AskUserQuestion` payload has no slot for a stable option key, so
there is nowhere for the hook to read one from. Even `source:"agent"` rows log
labels: 2 of 158 happen to match a `StandardOption`. **A perfect registry would
still yield 0.5 for every dimension.**
Gate 1 has its own cause: `scripts/resolvers/question-tuning.ts:30` tells every
skill to choose an id "from `` **or** `{skill}-{slug}`". That escape
hatch is what everyone uses — 333 of 983 distinct ids are auto-generated
`hook-`, meaning a third of questions carried no `` marker at
all. #2489 made the registry path absolute, which was necessary but not
sufficient: coverage is still 0.4% of rows.
## Related: the accumulator saturates in ~8 sessions
`normalizeToDimensionValue` is a sigmoid over an **accumulated** total, and
per-event deltas are unbounded. From the run above, 5 events reached 0.673, and
the observed total matches `1/(1+e^(-3t))` exactly, so:
```
5 events at +0.05 -> 0.679 20 -> 0.953
10 events at +0.05 -> 0.818 40 -> 0.998
```
One `/plan-eng-review` produces about 5. So the day gate coverage is fixed,
dimensions will pin at 1.0 — as uninformative as the 0.5 they replaced. Fixing
coverage without fixing this trades one silent failure for another.
## The failure is invisible by construction
`normalizeToDimensionValue(0)` is exactly 0.5, and `profile.inferred` records
`values`, `sample_size` and `diversity` but no per-dimension provenance. So
"no evidence" and "perfectly balanced evidence" print the same number. That is
why this sat unnoticed for two months with `sample_size: 1286`,
`skills_covered: 15` and `days_span: 20` all looking healthy beside five 0.5s.
## Suggested direction
1. **Give gate 3 a source of stable keys**, or drop the `StandardOption`
contract and key `SIGNAL_MAP` on something the hook can actually observe.
This is the design call; everything else is downstream of it.
2. **Record per-dimension `contributing_events`** in `profile.inferred` so a
dead dimension is visible rather than inferred. (PR attached does this.)
3. **Make readiness and mismatch logic consume those counts.** Today
`--check-mismatch` and the `CALIBRATED` gate in `plan-tune/SKILL.md:1237`
will happily report drift for a dimension with zero contributors.
4. **Bound per-event accumulation** so no dimension can saturate.
5. **Reconsider the `{skill}-{slug}` escape hatch**, or have unmapped ids carry
an explicit dimension hint. `scripts/one-way-doors.ts` is the in-repo
precedent for a fallback classifier over unmapped ids.
`TODOS.md` E3 (`:1301-1322`) and E6 (`:1383`) are both explicitly blocked on
"Calibrated profile (>= 20 events, 3+ skills, 7+ days span)". Those gates pass
today; the profile still is not calibrated.
## Reproduce
```bash
gstack-developer-profile --derive # DERIVE: ok — N events, ...
gstack-developer-profile --profile # inferred.values: all 0.5
gstack-developer-profile --trace autonomy # TRACE: no events contribute
```
Then answer one AskUserQuestion logged with a registered id and a stable option
key, re-run `--derive`, and watch that dimension move.
Contributor guide
Research direction
Run the three gstack-developer-profile reproduction commands, then read bin/gstack-developer-profile around lines 355, 356, and 423, scripts/resolvers/question-tuning.ts:30, and plan-tune/SKILL.md:1237. Determine the stable-key or observable-value design and verify that real AskUserQuestion events contribute, zero-contributor dimensions are visible, and accumulation cannot silently saturate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- analytics, cli, tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100