garrytan / garrytan/gstack

plan-tune question-log hook records answers as `__unknown__` on native AskUserQuestion; `followed_recommendation` always false

Open
#2,206 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133k
Forks
19.9k
Avg merge
18h 46m
Merged PRs (30d)
26

Description

## Summary

The plan-tune `PostToolUse` hook (`hosts/claude/hooks/question-log-hook.ts`) does not capture the user's actual answer on Claude Code's native `AskUserQuestion`. Logged events get `user_choice: "__unknown__"`.

There is a second, independent bug hiding behind the first: even when `user_choice` is extracted correctly, `followed_recommendation` still computes to `false` for every answer, because the two sides of the comparison are normalized differently.

Net effect: `/plan-tune`'s learning signal is not just missing, it is **inverted** — a user who always follows the recommendation accumulates a log that says they never do.

## Environment

- gstack `v1.58.5.0` (commit `11de390`)
- Claude Code, native (non-MCP) `AskUserQuestion`, `permission_mode: default`
- Linux, `bun` runtime
- Hooks installed via the documented `gstack-settings-hook add-event` commands printed by `./setup`

## Bug 1 — `extractUserChoices` reads the wrong key

`question-log-hook.ts` probes for two shapes:

```ts
// Shape A: { answers: [{option_label, free_text?}] }
if (Array.isArray(rec.answers)) { ... }
// Shape B: { questions: [{user_answer}] }
if (Array.isArray(rec.questions)) { ... }
```

The real `tool_response` (captured by teeing hook stdin) is:

```json
{
"tool_response": {
"questions": [
{
"question": "How should I report this upstream?",
"header": "Report form",
"options": [
{ "label": "Issue only (Recommended)", "description": "..." },
{ "label": "Issue plus a PR", "description": "..." }
],
"multiSelect": false
}
],
"answers": {
"How should I report this upstream?": "Issue only (Recommended)"
},
"annotations": {}
}
}
```

Two things go wrong:

1. `answers` **is present, but it is an object keyed by question text**, not an array. So `Array.isArray(rec.answers)` is `false` and Shape A is skipped.
2. `questions` **is an array**, so Shape B matches — but its elements are the question *definitions* echoed back, carrying `question` / `header` / `options` / `multiSelect`. There is no `user_answer`, `answer`, or `choice` key. Every element falls through to `'__unknown__'`.

So the `questions` branch shadows the branch that would have found the data. The stringify fallback (`__response-shape-unknown:`) never fires, which is why this reads as a silent data bug rather than an obvious shape mismatch.

Note `tool_input` also carries the same populated `answers` object, so the answer is available from two places.

### Suggested fix

Probe the object-keyed `answers` map **before** the `questions` array branch, matching each question by its text:

```ts
// Shape A′: { answers: { "": "" } }
if (rec.answers && typeof rec.answers === 'object' && !Array.isArray(rec.answers)) {
const map = rec.answers as Record;
for (const q of questionsFromInput) {
const choice = map[q.question ?? ''] ?? '__unknown__';
out.push({ choice });
}
return out;
}
```

This needs the question text from `tool_input.questions` to key the lookup, so `extractUserChoices` has to receive it rather than only `questionCount`.

## Bug 2 — `followed_recommendation` compares a raw label against a stripped one

`bin/gstack-question-log:173`:

```js
j.followed_recommendation = j.user_choice === j.recommended;
```

But `recommended` has already had the marker suffix stripped, at `question-log-hook.ts:144`:

```ts
if (labelMatches.length === 1)
return labelMatches[0].replace(RECOMMENDED_LABEL_RE, '').trim(); // "Issue only"
```

while `user_choice` is the raw option label the user actually picked, suffix intact: `"Issue only (Recommended)"`.

Strict equality therefore fails **exactly when the user follows the recommendation** — the one case the field exists to detect.

Reproducible today, independent of Bug 1, by calling the logger directly:

```bash
gstack-question-log '{"skill":"t","question_id":"x","question_summary":"q","options_count":2,
"user_choice":"Issue only (Recommended)","recommended":"Issue only","source":"hook"}'
```

Logged record:

```json
{ "user_choice": "Issue only (Recommended)",
"recommended": "Issue only",
"followed_recommendation": false }
```

### Suggested fix

Normalize both sides before comparing — strip `/\(recommended\)\s*$/i` from `user_choice` too, or compare with a shared `normalizeLabel()` helper. Doing this in `gstack-question-log` covers the `question-preference-hook` auto-decide path as well, which writes `user_choice` and `recommended` from the same already-stripped value and so is unaffected either way.

## Impact

Scope of what I actually observed: native `AskUserQuestion` under Claude Code, on gstack `v1.58.5.0`. I have **not** tested the `mcp__*__AskUserQuestion` variant or older Claude Code builds, and the hook's own comments note the response shape varies by variant — so I can't say how far this generalizes. On this configuration, though, it reproduces on every question.

- `question-log.jsonl` events written on this configuration all carry `user_choice: "__unknown__"`.
- Any `/plan-tune` analysis keyed on `user_choice` or `followed_recommendation` is drawing on empty or inverted data.
- Bug 2 is independent of variant: it lives in `bin/gstack-question-log` and will remain latent after Bug 1 is fixed, presenting as "the user ignores my recommendations."
- Affected logs cannot be repaired retroactively — the chosen label was never recorded.

## Reproduction

1. Install the plan-tune hooks as `./setup` instructs.
2. Trigger any `AskUserQuestion` and pick the option labelled `(Recommended)`.
3. `tail -1 ~/.gstack/projects//question-log.jsonl`.

Observed: `"user_choice": "__unknown__"`, `"followed_recommendation": false`.
Expected: the chosen option label, and `"followed_recommendation": true`.

Contributor guide

Open the contributing guide

Research direction

Start in hosts/claude/hooks/question-log-hook.ts at extractUserChoices and the recommendation handling, then inspect bin/gstack-question-log around line 173. Reproduce with the direct logger command and a native AskUserQuestion; done when the selected label is recorded and followed_recommendation is true for the recommended option.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.