AOSSIE-Org / AOSSIE-Org/EduAid

[ENHANCEMENT]: Add Output Validation for LLM-generated Question Responses

Aberta
#656 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
JavaScript
Estrelas
171
Forks
423
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

### Feature and its Use Cases

### Problem

LLM-based endpoints (`/get_mcq_llm`, `/get_boolq_llm`, `/get_shortq_llm`, `/get_problems_llm`) return generated questions without validating their structure or content.

As a result, the API may return HTTP 200 responses containing:

- Missing required fields (e.g., `options`, `answer`)
- Empty or malformed values
- Incomplete question sets
- Structurally inconsistent data

These issues are not detected before sending the response, leading to **silent failures** and unreliable outputs.

---

### Why This Matters

- Frontend may break due to undefined fields
- Users receive incomplete or unusable quizzes
- Debugging becomes difficult due to lack of validation
- Reduces overall reliability of LLM-based features

---

### Proposed Solution

Introduce a **lightweight output validation layer** for LLM-generated responses.

This layer should validate each generated question before it is returned by the API.

---

### Validation Rules

**MCQ**
- `question`: non-empty string
- `options`: non-empty list
- `correct_answer`: non-empty

**Short Questions**
- `question`: non-empty string
- `answer`: non-empty string

**Boolean Questions**
- `question`: non-empty string
- `answer`: must be `True` or `False`

---

### Expected Behavior

- Invalid questions are filtered out
- API responses contain only valid data
- No malformed or incomplete structures are returned
- Existing API behavior remains unchanged

---

### Scope

**In Scope**
- Validation of LLM-generated outputs
- Integration after parsing and before API response

**Out of Scope**
- Retry logic
- Pipeline/architecture changes
- Frontend changes
- Advanced schema validation systems

---

### Suggested Implementation

- Add a new module: `backend/Generator/output_validator.py`
- Implement simple validation functions per question type
- Integrate validation inside `llm_generator.py` before returning responses

---

### Impact

- Improves reliability of LLM endpoints
- Prevents silent failures
- Ensures consistent API contracts
- Enhances user experience

---

### Additional Context

### Additional Context

Input/request validation exists in parts of the system, but **LLM output validation is currently missing**, creating a gap in response correctness.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.