AOSSIE-Org / AOSSIE-Org/EduAid

[ENHANCEMENT]: Add Output Validation for LLM-generated Question Responses

Abierto
#656 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
JavaScript
Estrellas
171
Forks
423
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Feature and its Use Cases

### Problem

LLM-based endpoints (`/get_mcq_llm`, `/get_boolq_llm`, `/get_shortq_llm`, `/get_problems_llm`) return generated questions without validating their structure or content.

As a result, the API may return HTTP 200 responses containing:

- Missing required fields (e.g., `options`, `answer`)
- Empty or malformed values
- Incomplete question sets
- Structurally inconsistent data

These issues are not detected before sending the response, leading to **silent failures** and unreliable outputs.

---

### Why This Matters

- Frontend may break due to undefined fields
- Users receive incomplete or unusable quizzes
- Debugging becomes difficult due to lack of validation
- Reduces overall reliability of LLM-based features

---

### Proposed Solution

Introduce a **lightweight output validation layer** for LLM-generated responses.

This layer should validate each generated question before it is returned by the API.

---

### Validation Rules

**MCQ**
- `question`: non-empty string
- `options`: non-empty list
- `correct_answer`: non-empty

**Short Questions**
- `question`: non-empty string
- `answer`: non-empty string

**Boolean Questions**
- `question`: non-empty string
- `answer`: must be `True` or `False`

---

### Expected Behavior

- Invalid questions are filtered out
- API responses contain only valid data
- No malformed or incomplete structures are returned
- Existing API behavior remains unchanged

---

### Scope

**In Scope**
- Validation of LLM-generated outputs
- Integration after parsing and before API response

**Out of Scope**
- Retry logic
- Pipeline/architecture changes
- Frontend changes
- Advanced schema validation systems

---

### Suggested Implementation

- Add a new module: `backend/Generator/output_validator.py`
- Implement simple validation functions per question type
- Integrate validation inside `llm_generator.py` before returning responses

---

### Impact

- Improves reliability of LLM endpoints
- Prevents silent failures
- Ensures consistent API contracts
- Enhances user experience

---

### Additional Context

### Additional Context

Input/request validation exists in parts of the system, but **LLM output validation is currently missing**, creating a gap in response correctness.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.