AOSSIE-Org / AOSSIE-Org/EduAid

[FEATURE]: Centralize device management and model loading across generator classes

Abierto
#501 2 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
JavaScript
Estrellas
171
Forks
423
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Feature and its Use Cases

## Summary

While exploring the backend architecture, I noticed that model loading and device detection logic are currently duplicated across multiple generator classes (MCQGenerator, ShortQGenerator, BoolQGenerator, ParaphraseGenerator, AnswerPredictor, QuestionGenerator, etc.).

Each class independently:

- Detects CUDA availability
- Creates its own `torch.device`
- Moves its model to that device
- In some cases, clears CUDA cache

Additionally, certain transformer checkpoints (e.g., T5-based models) appear to be loaded separately in multiple classes.

---

## Why This Could Be Improved

Although models are correctly instantiated once at application startup (which is good for performance), the current structure results in:

- Repeated device-detection logic across classes
- Increased risk of inconsistencies (e.g., missing `.to(device)` in certain scenarios)
- Multiple independent loads of similar transformer checkpoints
- Harder future scaling if GPU handling or model lifecycle changes

Centralizing this logic could improve:

- Maintainability
- Device consistency
- Memory efficiency (shared model instances where applicable)
- Future extensibility (multi-GPU support, lazy loading, batching, etc.)

---

## Proposed Direction (Open for Discussion)

A possible minimal approach:

1. Introduce a shared device utility (single source of truth for `torch.device`)
2. Optionally introduce a lightweight model registry/factory for shared checkpoints
3. Refactor generator classes to receive device/model references instead of redefining them

This would keep existing API behavior unchanged while improving internal structure.

---

## Use Cases

This change would make it easier to:

- Add new generator classes without duplicating hardware logic
- Modify GPU/CPU behavior from a single location
- Prevent device mismatch errors
- Optimize memory usage by sharing model instances
- Extend the system for future hardware scaling or deployment improvements

---

I would appreciate maintainer feedback on whether this direction aligns with the project's roadmap before proceeding with any refactoring work.

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.