Capgemini / Capgemini/gov-prototype-by-prompt
The AI cannot generate `next_question_value` correctly for longer forms
- Dominant language
- TypeScript
- Stars
- 20
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
When generating a long form (>10 questions) from scratch, the AI struggles to generate the correct `next_question_value` for each question. This is likely to be because the AI cannot keep count.
### Steps to reproduce
This can be reproduced by asking the AI to generate a form with many questions.
Some open-ended prompts during testing also had a large number of questions:
- "Create a checklist for preparing the christmas meal" had 12 questions.
- "Create a form to apply for asylum in the UK by capturing all the information required by decision makers in the Home Office." had 20 questions.
- "Create a service to take your dog, cat or ferret to Northern Ireland" had 11 questions.
### How should the bug be fixed?
- Allow the AI to leave `next_question_value` as null to go to the next question, and fill in the value automatically.
- Always assume the last question points to the end, fill this in automatically regardless of what the AI generates.
- Improve the system prompt to be clear to the LLM that question numbers count from 1.
- If `next_question_value` points to the current question, automatically fix this to point to the next one.
- E.g., for Q3 to point to the next question then `next_question_value` should be `4`. But if the LLM mistakenly counts from 0 it would think the next question is number `3`.
- Consider how this could work for branching questions, if at all.
A more complex approach could be to split the generation into multiple smaller steps. For example, have one step to generate all the questions, and another to put them in the right order.
### Device, OS and browser used
_No response_
### Additional context (if applicable)
_No response_
### Screenshots (if applicable)
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.