AOSSIE-Org / AOSSIE-Org/EduAid

[ENHANCEMENT]: Add Output Validation for LLM-generated Question Responses

オープン
#656 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
JavaScript
スター
171
フォーク
423
PR マージ指標
30日以内にマージされた PR はありません

説明

### Feature and its Use Cases

### Problem

LLM-based endpoints (`/get_mcq_llm`, `/get_boolq_llm`, `/get_shortq_llm`, `/get_problems_llm`) return generated questions without validating their structure or content.

As a result, the API may return HTTP 200 responses containing:

- Missing required fields (e.g., `options`, `answer`)
- Empty or malformed values
- Incomplete question sets
- Structurally inconsistent data

These issues are not detected before sending the response, leading to **silent failures** and unreliable outputs.

---

### Why This Matters

- Frontend may break due to undefined fields
- Users receive incomplete or unusable quizzes
- Debugging becomes difficult due to lack of validation
- Reduces overall reliability of LLM-based features

---

### Proposed Solution

Introduce a **lightweight output validation layer** for LLM-generated responses.

This layer should validate each generated question before it is returned by the API.

---

### Validation Rules

**MCQ**
- `question`: non-empty string
- `options`: non-empty list
- `correct_answer`: non-empty

**Short Questions**
- `question`: non-empty string
- `answer`: non-empty string

**Boolean Questions**
- `question`: non-empty string
- `answer`: must be `True` or `False`

---

### Expected Behavior

- Invalid questions are filtered out
- API responses contain only valid data
- No malformed or incomplete structures are returned
- Existing API behavior remains unchanged

---

### Scope

**In Scope**
- Validation of LLM-generated outputs
- Integration after parsing and before API response

**Out of Scope**
- Retry logic
- Pipeline/architecture changes
- Frontend changes
- Advanced schema validation systems

---

### Suggested Implementation

- Add a new module: `backend/Generator/output_validator.py`
- Implement simple validation functions per question type
- Integrate validation inside `llm_generator.py` before returning responses

---

### Impact

- Improves reliability of LLM endpoints
- Prevents silent failures
- Ensures consistent API contracts
- Enhances user experience

---

### Additional Context

### Additional Context

Input/request validation exists in parts of the system, but **LLM output validation is currently missing**, creating a gap in response correctness.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。