AOSSIE-Org / AOSSIE-Org/EduAid

[FEATURE] Add Content-Based Inference Result Caching to Avoid Repeated Model Computation

Aperta
#558 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
JavaScript
Stelle
171
Fork
423
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Feature and its Use Cases

## The Problem Context

Currently, every request to the EduAid backend triggers full transformer model inference even if the exact same input text has already been processed earlier.

The backend runs large NLP models such as:
- T5-large
- T5-base
- DistilBERT

These models are computationally expensive and take several seconds to generate results.

In real educational scenarios, the same input text is often reused. Examples include:
- Teachers generating questions multiple times while preparing lessons
- Students retrying question generation on the same passage
- Common educational materials such as textbook excerpts or Wikipedia articles being reused
- Repeated queries during testing or demonstrations

Since there is currently no caching mechanism, the system recomputes results unnecessarily, which leads to:
- Increased response latency
- Higher GPU/CPU usage
- Reduced backend scalability

## Proposed Feature

Introduce content-based inference result caching so that repeated requests with identical input can reuse previously generated results.

The caching system would work as follows:

1. Generate a cache key using a hash of relevant request parameters.

Example:

SHA256(input_text + endpoint_name + generation_parameters)

2. Before running model inference:
- Check if the cache key already exists.

3. If a cached result exists:
- Return the stored result immediately.

4. If no cached result exists:
- Run the normal inference pipeline
- Store the generated result in the cache
- Return the result to the user

## Example Workflow

Current behavior:

Input Text

Model Inference

Return Result

Proposed behavior:

Input Text

Generate Cache Key

Check Cache
├── Cache Hit → Return Cached Result
└── Cache Miss → Run Model → Store Result → Return Result

## Potential Implementation

A lightweight caching layer could be implemented using a decorator pattern.

Example:

@cache_result(ttl=86400)
def generate_mcq(payload):
...

Key considerations:

- Cache key based on input text and generation parameters
- Configurable TTL (for example 24 hours)
- Configurable cache size limit
- Optional LRU eviction policy
- Ability to disable caching through configuration if needed

## Expected Benefits

- Faster responses for repeated queries
- Reduced GPU/CPU inference load
- Improved backend scalability
- Better user experience for repeated educational content

Cache hits could reduce response time from several seconds to a few milliseconds.

### Additional Context

## Additional Context

This feature can be implemented as an optional backend optimization without modifying the existing inference logic.

If caching is disabled or unavailable, the system should continue operating exactly as it currently does.

This makes the enhancement safe, incremental, and backward compatible with the existing architecture.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.