AOSSIE-Org / AOSSIE-Org/EduAid
[Enhancement] Add Optional Model Quantization to Reduce Memory Usage and Improve Inference Performance
- 主要言語
- JavaScript
- スター
- 171
- フォーク
- 423
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
### Feature and its Use Cases
### Feature Description
The EduAid backend currently loads several large NLP models in full FP32 precision, which results in significant memory consumption and increased inference latency.
Some of the major models currently used include:
- T5-large for MCQ generation
- T5-base for question generation
- DistilBERT for NLI-based boolean answer prediction
- SpaCy for NLP processing
- Sense2Vec for distractor generation
These models are currently loaded in full precision which increases:
- Memory usage
- Server resource consumption
- Cold start latency
- Risk of Out-of-Memory (OOM) errors when handling multiple requests
### Proposed Enhancement
Introduce **optional model quantization and precision optimization** during model loading to reduce memory footprint and improve inference speed.
This can be implemented inside:
```
backend/Generator/main.py
```
Specifically in the:
```
ModelManager.load_model()
```
method.
### Possible Implementation
1. Apply **dynamic INT8 quantization** for transformer models when running on CPU.
Example:
```python
torch.quantization.quantize_dynamic(
model,
{torch.nn.Linear},
dtype=torch.qint8
)
```
2. Use **FP16 precision when CUDA is available**.
Example:
```python
model = model.half()
```
3. Add a configuration flag such as:
```
ENABLE_MODEL_QUANTIZATION = True
```
to allow enabling or disabling this behavior.
### Expected Benefits
- Reduce memory consumption by ~40–60%
- Faster inference for transformer models
- Lower server resource usage
- Better scalability for concurrent requests
- Reduced deployment costs
### Why This Matters
Currently the backend may load multiple large models simultaneously which can lead to memory usage exceeding **5–7GB** in some environments.
Applying quantization could reduce this significantly while maintaining acceptable model accuracy for question generation tasks.
### Files Likely Affected
```
backend/Generator/main.py
backend/server.py
```
### Additional Context
### Additional Context
This improvement focuses on optimizing model resource usage without changing the existing functionality of the quiz generation pipeline.
The change would be fully backward compatible and could be implemented behind a feature flag so that deployments can enable or disable quantization depending on the available hardware.
Models that would benefit most from this improvement include:
- T5-large models used in MCQ generation
- T5-base models used for question generation
- DistilBERT-based NLI models used for boolean answer prediction
Other models such as SpaCy and Sense2Vec may not significantly benefit from quantization but can still remain part of the existing pipeline.
This enhancement could significantly improve backend performance and make the system more efficient for production deployments.
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。