Capgemini / Capgemini/gov-prototype-by-prompt

The AI cannot generate `next_question_value` correctly for longer forms

Open
#144 0 comments 0 reactions 1 assignee Claimed by @cmenon12 View on GitHub
Dominant language
TypeScript
Stars
20
Forks
4
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

When generating a long form (>10 questions) from scratch, the AI struggles to generate the correct `next_question_value` for each question. This is likely to be because the AI cannot keep count.

### Steps to reproduce

This can be reproduced by asking the AI to generate a form with many questions.

Some open-ended prompts during testing also had a large number of questions:
- "Create a checklist for preparing the christmas meal" had 12 questions.
- "Create a form to apply for asylum in the UK by capturing all the information required by decision makers in the Home Office." had 20 questions.
- "Create a service to take your dog, cat or ferret to Northern Ireland" had 11 questions.

### How should the bug be fixed?

- Allow the AI to leave `next_question_value` as null to go to the next question, and fill in the value automatically.
- Always assume the last question points to the end, fill this in automatically regardless of what the AI generates.
- Improve the system prompt to be clear to the LLM that question numbers count from 1.
- If `next_question_value` points to the current question, automatically fix this to point to the next one.
- E.g., for Q3 to point to the next question then `next_question_value` should be `4`. But if the LLM mistakenly counts from 0 it would think the next question is number `3`.
- Consider how this could work for branching questions, if at all.

A more complex approach could be to split the generation into multiple smaller steps. For example, have one step to generate all the questions, and another to put them in the right order.

### Device, OS and browser used

_No response_

### Additional context (if applicable)

_No response_

### Screenshots (if applicable)

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.