AOSSIE-Org / AOSSIE-Org/EduAid

[Enhancement]: Implement Global Lazy Loading for Transformer Models to Reduce Memory Usage

Offen
#525 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
JavaScript
Sterne
171
Forks
425
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Feature and its Use Cases

Currently, multiple generator classes in the backend (MCQGenerator, ShortQGenerator, BoolQGenerator, AnswerPredictor, etc.) independently load transformer models using `from_pretrained()`.

This can lead to redundant model initialization and higher memory usage because the same models may be loaded multiple times across different components.

Proposed Feature:
Introduce a centralized lazy-loading model cache (for example, a ModelRegistry or model_cache module) that ensures each pretrained transformer model is loaded only once and reused across the application.

How it would work:
- Create a shared model registry or cache.
- Load a model only when it is first requested (lazy loading).
- Reuse the same instance across generator classes.

Benefits for users and developers:
- Reduced memory usage.
- Faster backend initialization.
- Improved scalability when handling concurrent requests.
- Cleaner backend architecture.

Use Cases:
1. When the backend initializes multiple generator classes that rely on the same transformer models.
2. When multiple API endpoints trigger model loading simultaneously.
3. When deploying EduAid on servers with limited memory resources.

### Additional Context

This enhancement would improve the backend performance and maintainability of EduAid by preventing redundant model loading and encouraging a more centralized architecture for managing transformer models.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.