AOSSIE-Org / AOSSIE-Org/EduAid

[FEATURE]: Centralize device management and model loading across generator classes

Offen
#501 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
JavaScript
Sterne
171
Forks
423
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Feature and its Use Cases

## Summary

While exploring the backend architecture, I noticed that model loading and device detection logic are currently duplicated across multiple generator classes (MCQGenerator, ShortQGenerator, BoolQGenerator, ParaphraseGenerator, AnswerPredictor, QuestionGenerator, etc.).

Each class independently:

- Detects CUDA availability
- Creates its own `torch.device`
- Moves its model to that device
- In some cases, clears CUDA cache

Additionally, certain transformer checkpoints (e.g., T5-based models) appear to be loaded separately in multiple classes.

---

## Why This Could Be Improved

Although models are correctly instantiated once at application startup (which is good for performance), the current structure results in:

- Repeated device-detection logic across classes
- Increased risk of inconsistencies (e.g., missing `.to(device)` in certain scenarios)
- Multiple independent loads of similar transformer checkpoints
- Harder future scaling if GPU handling or model lifecycle changes

Centralizing this logic could improve:

- Maintainability
- Device consistency
- Memory efficiency (shared model instances where applicable)
- Future extensibility (multi-GPU support, lazy loading, batching, etc.)

---

## Proposed Direction (Open for Discussion)

A possible minimal approach:

1. Introduce a shared device utility (single source of truth for `torch.device`)
2. Optionally introduce a lightweight model registry/factory for shared checkpoints
3. Refactor generator classes to receive device/model references instead of redefining them

This would keep existing API behavior unchanged while improving internal structure.

---

## Use Cases

This change would make it easier to:

- Add new generator classes without duplicating hardware logic
- Modify GPU/CPU behavior from a single location
- Prevent device mismatch errors
- Optimize memory usage by sharing model instances
- Extend the system for future hardware scaling or deployment improvements

---

I would appreciate maintainer feedback on whether this direction aligns with the project's roadmap before proceeding with any refactoring work.

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.