AOSSIE-Org / AOSSIE-Org/Devr.AI

ENHANCEMENT:Reduce unnecessary LLM API calls

Đang mở
#241 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
102
Fork
137
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

## 🌟 Feature Description

Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.

This feature will:

* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls

---

## 🔍 Problem Statement

Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.

This leads to:

* Unnecessary API usage increasing
* Increased latency
* Higher operational costs

### Current Behavior

```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```

Every incoming message triggers the LLM, regardless of whether it is:

* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment

---

## 🎯 Expected Outcome

After this enhancement:

* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly

This will reduce API calls, lower costs, and make the system faster and more scalable.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.