Capgemini / Capgemini/gov-prototype-by-prompt

The local LLM is incapable of generating a valid prototype

Open
#154 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
20
Forks
4
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

When using the local LLM `ai/gpt-oss`, the LLM cannot generate a valid prototype and fails every time.

- It cannot correctly generate the `next_question_value` that points to the number of the next question.
- It uses the wrong `answer_type`, e.g., "text" for an address question
- It adds extraneous properties, e.g., adds options to questions that don't need them.
- It adds an empty "submit" question at the end.

### Steps to reproduce

Run the Docker Compose script with the local LLM `ai/gpt-oss`.

### How should the bug be fixed?

Options include:

- Try out different models.
- Improve the prompt & schema to coax the model. But need to be careful about tuning them for a local LLM to the detriment of the Azure LLM.
- Split the generation step into multiple steps (e.g., generate the questions, then put them in order). This is a larger piece of work but likely necessary anyway to generate larger prototypes

### Device, OS and browser used

_No response_

### Additional context (if applicable)

Follow-on from #148 and #152.

### Screenshots (if applicable)

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with Docker Compose and the local ai/gpt-oss model, then inspect the prompt and schema used for prototype generation. Compare the output against the required next_question_value, answer_type, question properties, and terminal submit behavior; done means valid prototypes are produced without regressing Azure LLM behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
ai
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.