AOSSIE-Org / AOSSIE-Org/EduAid

[FEATURE]: Centralize device management and model loading across generator classes

Aperta
#501 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
JavaScript
Stelle
171
Fork
423
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Feature and its Use Cases

## Summary

While exploring the backend architecture, I noticed that model loading and device detection logic are currently duplicated across multiple generator classes (MCQGenerator, ShortQGenerator, BoolQGenerator, ParaphraseGenerator, AnswerPredictor, QuestionGenerator, etc.).

Each class independently:

- Detects CUDA availability
- Creates its own `torch.device`
- Moves its model to that device
- In some cases, clears CUDA cache

Additionally, certain transformer checkpoints (e.g., T5-based models) appear to be loaded separately in multiple classes.

---

## Why This Could Be Improved

Although models are correctly instantiated once at application startup (which is good for performance), the current structure results in:

- Repeated device-detection logic across classes
- Increased risk of inconsistencies (e.g., missing `.to(device)` in certain scenarios)
- Multiple independent loads of similar transformer checkpoints
- Harder future scaling if GPU handling or model lifecycle changes

Centralizing this logic could improve:

- Maintainability
- Device consistency
- Memory efficiency (shared model instances where applicable)
- Future extensibility (multi-GPU support, lazy loading, batching, etc.)

---

## Proposed Direction (Open for Discussion)

A possible minimal approach:

1. Introduce a shared device utility (single source of truth for `torch.device`)
2. Optionally introduce a lightweight model registry/factory for shared checkpoints
3. Refactor generator classes to receive device/model references instead of redefining them

This would keep existing API behavior unchanged while improving internal structure.

---

## Use Cases

This change would make it easier to:

- Add new generator classes without duplicating hardware logic
- Modify GPU/CPU behavior from a single location
- Prevent device mismatch errors
- Optimize memory usage by sharing model instances
- Extend the system for future hardware scaling or deployment improvements

---

I would appreciate maintainer feedback on whether this direction aligns with the project's roadmap before proceeding with any refactoring work.

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.