AOSSIE-Org / AOSSIE-Org/EduAid
[FEATURE] Add Content-Based Inference Result Caching to Avoid Repeated Model Computation
- Ngôn ngữ chính
- JavaScript
- Star
- 171
- Fork
- 423
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Feature and its Use Cases
## The Problem Context
Currently, every request to the EduAid backend triggers full transformer model inference even if the exact same input text has already been processed earlier.
The backend runs large NLP models such as:
- T5-large
- T5-base
- DistilBERT
These models are computationally expensive and take several seconds to generate results.
In real educational scenarios, the same input text is often reused. Examples include:
- Teachers generating questions multiple times while preparing lessons
- Students retrying question generation on the same passage
- Common educational materials such as textbook excerpts or Wikipedia articles being reused
- Repeated queries during testing or demonstrations
Since there is currently no caching mechanism, the system recomputes results unnecessarily, which leads to:
- Increased response latency
- Higher GPU/CPU usage
- Reduced backend scalability
## Proposed Feature
Introduce content-based inference result caching so that repeated requests with identical input can reuse previously generated results.
The caching system would work as follows:
1. Generate a cache key using a hash of relevant request parameters.
Example:
SHA256(input_text + endpoint_name + generation_parameters)
2. Before running model inference:
- Check if the cache key already exists.
3. If a cached result exists:
- Return the stored result immediately.
4. If no cached result exists:
- Run the normal inference pipeline
- Store the generated result in the cache
- Return the result to the user
## Example Workflow
Current behavior:
Input Text
↓
Model Inference
↓
Return Result
Proposed behavior:
Input Text
↓
Generate Cache Key
↓
Check Cache
├── Cache Hit → Return Cached Result
└── Cache Miss → Run Model → Store Result → Return Result
## Potential Implementation
A lightweight caching layer could be implemented using a decorator pattern.
Example:
@cache_result(ttl=86400)
def generate_mcq(payload):
...
Key considerations:
- Cache key based on input text and generation parameters
- Configurable TTL (for example 24 hours)
- Configurable cache size limit
- Optional LRU eviction policy
- Ability to disable caching through configuration if needed
## Expected Benefits
- Faster responses for repeated queries
- Reduced GPU/CPU inference load
- Improved backend scalability
- Better user experience for repeated educational content
Cache hits could reduce response time from several seconds to a few milliseconds.
### Additional Context
## Additional Context
This feature can be implemented as an optional backend optimization without modifying the existing inference logic.
If caching is disabled or unavailable, the system should continue operating exactly as it currently does.
This makes the enhancement safe, incremental, and backward compatible with the existing architecture.
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.