AOSSIE-Org / AOSSIE-Org/EduAid

[ENHANCEMENT]: Add Output Validation for LLM-generated Question Responses

Đang mở
#656 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
JavaScript
Star
171
Fork
423
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Feature and its Use Cases

### Problem

LLM-based endpoints (`/get_mcq_llm`, `/get_boolq_llm`, `/get_shortq_llm`, `/get_problems_llm`) return generated questions without validating their structure or content.

As a result, the API may return HTTP 200 responses containing:

- Missing required fields (e.g., `options`, `answer`)
- Empty or malformed values
- Incomplete question sets
- Structurally inconsistent data

These issues are not detected before sending the response, leading to **silent failures** and unreliable outputs.

---

### Why This Matters

- Frontend may break due to undefined fields
- Users receive incomplete or unusable quizzes
- Debugging becomes difficult due to lack of validation
- Reduces overall reliability of LLM-based features

---

### Proposed Solution

Introduce a **lightweight output validation layer** for LLM-generated responses.

This layer should validate each generated question before it is returned by the API.

---

### Validation Rules

**MCQ**
- `question`: non-empty string
- `options`: non-empty list
- `correct_answer`: non-empty

**Short Questions**
- `question`: non-empty string
- `answer`: non-empty string

**Boolean Questions**
- `question`: non-empty string
- `answer`: must be `True` or `False`

---

### Expected Behavior

- Invalid questions are filtered out
- API responses contain only valid data
- No malformed or incomplete structures are returned
- Existing API behavior remains unchanged

---

### Scope

**In Scope**
- Validation of LLM-generated outputs
- Integration after parsing and before API response

**Out of Scope**
- Retry logic
- Pipeline/architecture changes
- Frontend changes
- Advanced schema validation systems

---

### Suggested Implementation

- Add a new module: `backend/Generator/output_validator.py`
- Implement simple validation functions per question type
- Integrate validation inside `llm_generator.py` before returning responses

---

### Impact

- Improves reliability of LLM endpoints
- Prevents silent failures
- Ensures consistent API contracts
- Enhances user experience

---

### Additional Context

### Additional Context

Input/request validation exists in parts of the system, but **LLM output validation is currently missing**, creating a gap in response correctness.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.