github / github/app

GPT models cannot follow skills that ask multiple questions in rounds

Offen
#3,110 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Keine Sprachdaten
Sterne
2.1k
Forks
153
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## What happened?

I use Matt Pocock’s [`grill-me` skill](https://github.com/mattpocock/skills), specifically the round-based workflow introduced in v1.2 and tested with v1.2.3.

The skill builds a design tree and asks the entire current “frontier” of independent decisions in one Markdown response:

```text
❓ Q1 — ...
➡️ Recommended answer

❓ Q2 — ...
➡️ Recommended answer
```

This is much more efficient than asking every independent question in a separate turn.

With GPT-5.6 Sol, GitHub Copilot App refuses this format because questions must use the single-question `ask_user` UI. The skill is therefore forced into sequential questioning and cannot follow its documented protocol.

The same skill works with Claude Sonnet 5 in GitHub Copilot App. Claude presents the complete frontier directly in chat and waits for one combined response.

## Steps to reproduce

1. Install Matt Pocock’s skills v1.2 or later.
2. Select GPT-5.6 Sol.
3. Invoke `/grill-me` for a design containing several independent decisions.
4. Ask it to present the current frontier as one round.
5. Observe that it uses, or insists on using, the single-question UI.
6. Repeat with Claude Sonnet 5.
7. Observe that Sonnet renders the complete question round directly in chat.

## Expected behavior

Skills should be able to define a multi-question round protocol consistently across models.

Possible solutions:

- Permit numbered questions in normal chat when explicitly required by an invoked skill.
- Extend `ask_user` to support several independently answerable questions in one round.

## Actual behavior

GPT models follow a Copilot App instruction requiring every question to use the single-question UI. This overrides the skill’s explicit round-based protocol.

Claude Sonnet 5 follows the same skill as written and presents multiple numbered questions in one chat response.

## Impact

Sequential UI prompts add unnecessary turns and undermine the primary efficiency improvement introduced by `grill-me` v1.2.

Behavior also changes depending on the selected model, although the same skill and Copilot App environment are used.

## Environment

- GitHub Copilot App 1.0.80
- Matt Pocock skills v1.2.3
- GPT-5.6 Sol: round-based workflow blocked by single-question policy
- Claude Sonnet 5: round-based workflow works

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by reproducing the round-based /grill-me workflow in GitHub Copilot App 1.0.80 with GPT-5.6 Sol and Claude Sonnet 5, using Matt Pocock’s skills v1.2.3. Done means the multi-question round protocol works consistently across models, either through normal chat or a multi-question ask_user UI.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
github
Bereich
ai, desktop
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.