AOSSIE-Org / AOSSIE-Org/Devr.AI

ENHANCEMENT:Reduce unnecessary LLM API calls

Aberta
#241 2 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
102
Forks
137
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

## 🌟 Feature Description

Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.

This feature will:

* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls

---

## 🔍 Problem Statement

Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.

This leads to:

* Unnecessary API usage increasing
* Increased latency
* Higher operational costs

### Current Behavior

```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```

Every incoming message triggers the LLM, regardless of whether it is:

* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment

---

## 🎯 Expected Outcome

After this enhancement:

* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly

This will reduce API calls, lower costs, and make the system faster and more scalable.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.