Capgemini / Capgemini/gov-prototype-by-prompt
The local LLM is incapable of generating a valid prototype
- Dominant language
- TypeScript
- Stars
- 20
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
When using the local LLM `ai/gpt-oss`, the LLM cannot generate a valid prototype and fails every time.
- It cannot correctly generate the `next_question_value` that points to the number of the next question.
- It uses the wrong `answer_type`, e.g., "text" for an address question
- It adds extraneous properties, e.g., adds options to questions that don't need them.
- It adds an empty "submit" question at the end.
### Steps to reproduce
Run the Docker Compose script with the local LLM `ai/gpt-oss`.
### How should the bug be fixed?
Options include:
- Try out different models.
- Improve the prompt & schema to coax the model. But need to be careful about tuning them for a local LLM to the detriment of the Azure LLM.
- Split the generation step into multiple steps (e.g., generate the questions, then put them in order). This is a larger piece of work but likely necessary anyway to generate larger prototypes
### Device, OS and browser used
_No response_
### Additional context (if applicable)
Follow-on from #148 and #152.
### Screenshots (if applicable)
_No response_
Contributor guide
Research direction
Reproduce the failure with Docker Compose and the local ai/gpt-oss model, then inspect the prompt and schema used for prototype generation. Compare the output against the required next_question_value, answer_type, question properties, and terminal submit behavior; done means valid prototypes are produced without regressing Azure LLM behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100