AOSSIE-Org / AOSSIE-Org/EduAid

[Enhancement]: Implement Global Lazy Loading for Transformer Models to Reduce Memory Usage

Aperta
#525 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
JavaScript
Stelle
171
Fork
425
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Feature and its Use Cases

Currently, multiple generator classes in the backend (MCQGenerator, ShortQGenerator, BoolQGenerator, AnswerPredictor, etc.) independently load transformer models using `from_pretrained()`.

This can lead to redundant model initialization and higher memory usage because the same models may be loaded multiple times across different components.

Proposed Feature:
Introduce a centralized lazy-loading model cache (for example, a ModelRegistry or model_cache module) that ensures each pretrained transformer model is loaded only once and reused across the application.

How it would work:
- Create a shared model registry or cache.
- Load a model only when it is first requested (lazy loading).
- Reuse the same instance across generator classes.

Benefits for users and developers:
- Reduced memory usage.
- Faster backend initialization.
- Improved scalability when handling concurrent requests.
- Cleaner backend architecture.

Use Cases:
1. When the backend initializes multiple generator classes that rely on the same transformer models.
2. When multiple API endpoints trigger model loading simultaneously.
3. When deploying EduAid on servers with limited memory resources.

### Additional Context

This enhancement would improve the backend performance and maintainability of EduAid by preventing redundant model loading and encouraging a more centralized architecture for managing transformer models.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.