AOSSIE-Org / AOSSIE-Org/Devr.AI
ENHANCEMENT:Reduce unnecessary LLM API calls
- Dominant language
- Python
- Stars
- 102
- Forks
- 137
- PR merge metrics
- No merged PRs in 30d
Description
### Is your feature request related to a problem?
- [x] Yes, it is related to a problem
### Describe the feature you'd like
## 🌟 Feature Description
Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.
This feature will:
* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls
---
## 🔍 Problem Statement
Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.
This leads to:
* Unnecessary API usage increasing
* Increased latency
* Higher operational costs
### Current Behavior
```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```
Every incoming message triggers the LLM, regardless of whether it is:
* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment
---
## 🎯 Expected Outcome
After this enhancement:
* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly
This will reduce API calls, lower costs, and make the system faster and more scalable.
### Record
- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.