AOSSIE-Org / AOSSIE-Org/Devr.AI

ENHANCEMENT:Reduce unnecessary LLM API calls

Abierto
#241 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
102
Forks
137
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

## 🌟 Feature Description

Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.

This feature will:

* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls

---

## 🔍 Problem Statement

Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.

This leads to:

* Unnecessary API usage increasing
* Increased latency
* Higher operational costs

### Current Behavior

```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```

Every incoming message triggers the LLM, regardless of whether it is:

* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment

---

## 🎯 Expected Outcome

After this enhancement:

* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly

This will reduce API calls, lower costs, and make the system faster and more scalable.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.