AOSSIE-Org / AOSSIE-Org/EduAid

[FEATURE] Add Content-Based Inference Result Caching to Avoid Repeated Model Computation

Open
#558 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
171
Forks
423
PR merge metrics
No merged PRs in 30d

Description

### Feature and its Use Cases

## The Problem Context

Currently, every request to the EduAid backend triggers full transformer model inference even if the exact same input text has already been processed earlier.

The backend runs large NLP models such as:
- T5-large
- T5-base
- DistilBERT

These models are computationally expensive and take several seconds to generate results.

In real educational scenarios, the same input text is often reused. Examples include:
- Teachers generating questions multiple times while preparing lessons
- Students retrying question generation on the same passage
- Common educational materials such as textbook excerpts or Wikipedia articles being reused
- Repeated queries during testing or demonstrations

Since there is currently no caching mechanism, the system recomputes results unnecessarily, which leads to:
- Increased response latency
- Higher GPU/CPU usage
- Reduced backend scalability

## Proposed Feature

Introduce content-based inference result caching so that repeated requests with identical input can reuse previously generated results.

The caching system would work as follows:

1. Generate a cache key using a hash of relevant request parameters.

Example:

SHA256(input_text + endpoint_name + generation_parameters)

2. Before running model inference:
- Check if the cache key already exists.

3. If a cached result exists:
- Return the stored result immediately.

4. If no cached result exists:
- Run the normal inference pipeline
- Store the generated result in the cache
- Return the result to the user

## Example Workflow

Current behavior:

Input Text

Model Inference

Return Result

Proposed behavior:

Input Text

Generate Cache Key

Check Cache
├── Cache Hit → Return Cached Result
└── Cache Miss → Run Model → Store Result → Return Result

## Potential Implementation

A lightweight caching layer could be implemented using a decorator pattern.

Example:

@cache_result(ttl=86400)
def generate_mcq(payload):
...

Key considerations:

- Cache key based on input text and generation parameters
- Configurable TTL (for example 24 hours)
- Configurable cache size limit
- Optional LRU eviction policy
- Ability to disable caching through configuration if needed

## Expected Benefits

- Faster responses for repeated queries
- Reduced GPU/CPU inference load
- Improved backend scalability
- Better user experience for repeated educational content

Cache hits could reduce response time from several seconds to a few milliseconds.

### Additional Context

## Additional Context

This feature can be implemented as an optional backend optimization without modifying the existing inference logic.

If caching is disabled or unavailable, the system should continue operating exactly as it currently does.

This makes the enhancement safe, incremental, and backward compatible with the existing architecture.

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.