microsoft / microsoft/PromptKit

Investigate: structured feedback classification as a cross-cutting protocol

Open
#167 0 comments 0 reactions 1 assignee View on GitHub

@abeltrano is already working on this.

Since Apr 2, 2026.

enhancement
Dominant language
JavaScript
Stars
105
Forks
23
PR merge metrics
No merged PRs in 30d

Description

Motivation

PR #166 added a feedback classification rule to the spec-extraction-workflow template (Phase 3) that requires the LLM to categorize user feedback as Editorial, Correction, or Finding — defaulting to Finding when uncertain. This prevented a real failure mode where domain-expert feedback was silently under-promoted to a cosmetic change.

The underlying problem — LLMs default to the lowest-impact interpretation of user input — is not specific to spec extraction. Any interactive template where the user provides feedback mid-workflow is susceptible.

Investigation Questions

  1. Which templates have interactive feedback loops? Candidates include:

    • interactive-design (Phase 1 reasoning loop)
    • extend-library (contributor guidance workflow)
    • engineering-workflow / collaborate-requirements-change (iterative change propagation)
    • evolve-protocol (protocol modification loop)
    • author-north-star (section-by-section drafting with user review)
    • author-presentation (multi-phase with user review)
    • maintenance-workflow (finding classification with user collaboration)
    • Any template using the iterative-refinement protocol
  2. Should this be a protocol rather than per-template guidance? If the pattern applies broadly, it may belong as:

    • A new guardrail protocol (e.g., feedback-classification) applied to all interactive templates
    • An addition to the existing iterative-refinement protocol (since that protocol already governs how changes are applied during feedback loops)
    • A section in the anti-hallucination protocol (since under-promoting feedback is a form of information loss)
  3. Does the classification taxonomy need to be richer for other contexts? The current three categories (Editorial / Correction / Finding) were designed for spec extraction. Other templates might need:

    • Scope change — user wants to expand or narrow the task
    • Constraint — user is adding a new constraint not previously stated
    • Preference — user is expressing a stylistic or approach preference (not a requirement)
    • Rejection — user is rejecting a proposed element entirely
  4. What is the interaction with existing protocols?

    • iterative-refinement already says 'justify every change' — should feedback classification be embedded there?
    • requirements-elicitation already decomposes user input — should it also classify the input type?
    • adversarial-falsification already has confidence classification — is there overlap?
  5. What form should it take?

    • A standalone guardrail protocol (protocols/guardrails/feedback-classification.md)
    • An extension to iterative-refinement protocol
    • Guidance in bootstrap.md for all interactive templates
    • A combination (protocol + bootstrap guidance)

Suggested Approach

  1. Audit all interactive templates and the iterative-refinement protocol for feedback handling gaps
  2. Draft a candidate protocol or protocol extension
  3. Test it against 2-3 real interactive sessions to validate the taxonomy
  4. Propose via extend-library template contribution workflow

Related

  • #165 — Original issue (spec-extraction-workflow specific fix)
  • #166 — PR implementing the spec-extraction-specific classification rule

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.