AOSSIE-Org / AOSSIE-Org/Devr.AI

ENHANCEMENT:Reduce unnecessary LLM API calls

未關閉
#241 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
102
分支
137
PR 合併指標
30 天內沒有已合併 PR

描述

### Is your feature request related to a problem?

- [x] Yes, it is related to a problem

### Describe the feature you'd like

## 🌟 Feature Description

Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.

This feature will:

* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls

---

## 🔍 Problem Statement

Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.

This leads to:

* Unnecessary API usage increasing
* Increased latency
* Higher operational costs

### Current Behavior

```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```

Every incoming message triggers the LLM, regardless of whether it is:

* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment

---

## 🎯 Expected Outcome

After this enhancement:

* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly

This will reduce API calls, lower costs, and make the system faster and more scalable.

### Record

- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。