AOSSIE-Org / AOSSIE-Org/Devr.AI

ENHANCEMENT:Reduce unnecessary LLM API calls

Open
#241 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
102
Forks
137
PR merge metrics
No merged PRs in 30d

Description

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

## 🌟 Feature Description

Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.

This feature will:

* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls

---

## 🔍 Problem Statement

Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.

This leads to:

* Unnecessary API usage increasing
* Increased latency
* Higher operational costs

### Current Behavior

```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```

Every incoming message triggers the LLM, regardless of whether it is:

* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment

---

## 🎯 Expected Outcome

After this enhancement:

* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly

This will reduce API calls, lower costs, and make the system faster and more scalable.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.