AOSSIE-Org / AOSSIE-Org/Devr.AI
ENHANCEMENT:Reduce unnecessary LLM API calls
- Ngôn ngữ chính
- Python
- Star
- 102
- Fork
- 137
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Is your feature request related to a problem?
- [x] Yes, it is related to a problem
### Describe the feature you'd like
## 🌟 Feature Description
Reduce unnecessary LLM API calls in the message classification system by adding **smart caching and simple pattern matching**.
This feature will:
* Detect **common messages** (e.g. greetings, thanks, acknowledgments) without calling the LLM
* Cache previous LLM classification results using an **LRU cache with TTL**
* Normalize messages (lowercase, trim spaces, etc.) to improve cache hits
* Track basic metrics to measure cache usage and saved LLM calls
---
## 🔍 Problem Statement
Currently, the `ClassificationRouter` makes an **LLM API call for every single Discord message**, even for very simple or repeated messages.
This leads to:
* Unnecessary API usage increasing
* Increased latency
* Higher operational costs
### Current Behavior
```python
async def should_process_message(self, message: str, context: Dict[str, Any] = None):
response = await self.llm.ainvoke([HumanMessage(content=triage_prompt)])
```
Every incoming message triggers the LLM, regardless of whether it is:
* A simple greeting like “hi”
* A repeated message
* A non-actionable acknowledgment
---
## 🎯 Expected Outcome
After this enhancement:
* Simple messages are handled using **pattern matching**
* Repeated messages reuse results from the **cache**
* LLM calls are made **only when truly needed**
* Overall performance and efficiency improve significantly
This will reduce API calls, lower costs, and make the system faster and more scalable.
### Record
- [x] I agree to follow this project's Code of Conduct
- [x] I want to work on implementing this feature
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.