AOSSIE-Org / AOSSIE-Org/EduAid
[FEATURE]: Centralize device management and model loading across generator classes
- Lingua principale
- JavaScript
- Stelle
- 171
- Fork
- 423
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### Feature and its Use Cases
## Summary
While exploring the backend architecture, I noticed that model loading and device detection logic are currently duplicated across multiple generator classes (MCQGenerator, ShortQGenerator, BoolQGenerator, ParaphraseGenerator, AnswerPredictor, QuestionGenerator, etc.).
Each class independently:
- Detects CUDA availability
- Creates its own `torch.device`
- Moves its model to that device
- In some cases, clears CUDA cache
Additionally, certain transformer checkpoints (e.g., T5-based models) appear to be loaded separately in multiple classes.
---
## Why This Could Be Improved
Although models are correctly instantiated once at application startup (which is good for performance), the current structure results in:
- Repeated device-detection logic across classes
- Increased risk of inconsistencies (e.g., missing `.to(device)` in certain scenarios)
- Multiple independent loads of similar transformer checkpoints
- Harder future scaling if GPU handling or model lifecycle changes
Centralizing this logic could improve:
- Maintainability
- Device consistency
- Memory efficiency (shared model instances where applicable)
- Future extensibility (multi-GPU support, lazy loading, batching, etc.)
---
## Proposed Direction (Open for Discussion)
A possible minimal approach:
1. Introduce a shared device utility (single source of truth for `torch.device`)
2. Optionally introduce a lightweight model registry/factory for shared checkpoints
3. Refactor generator classes to receive device/model references instead of redefining them
This would keep existing API behavior unchanged while improving internal structure.
---
## Use Cases
This change would make it easier to:
- Add new generator classes without duplicating hardware logic
- Modify GPU/CPU behavior from a single location
- Prevent device mismatch errors
- Optimize memory usage by sharing model instances
- Extend the system for future hardware scaling or deployment improvements
---
I would appreciate maintainer feedback on whether this direction aligns with the project's roadmap before proceeding with any refactoring work.
### Additional Context
_No response_
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.