AOSSIE-Org / AOSSIE-Org/Devr.AI
ENHANCEMENT:Reduce unnecessary LLM API calls
- Lenguaje dominante
- Python
- Estrellas
- 102
- Forks
- 137
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
### Is your feature request related to a problem?
- [x] Yes, it is related to a problem
### Describe the feature you'd like
## 🌟 Feature Description
Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.
This feature will:
* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls
---
## 🔍 Problem Statement
Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.
This leads to:
* Unnecessary API usage increasing
* Increased latency
* Higher operational costs
### Current Behavior
```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```
Every incoming message triggers the LLM, regardless of whether it is:
* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment
---
## 🎯 Expected Outcome
After this enhancement:
* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly
This will reduce API calls, lower costs, and make the system faster and more scalable.
### Record
- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.