google / google/adk-python

Native Litert-lm integration with local request queuing and KV cache management

Đang mở
#5,575 1 bình luận 1 reaction 1 người được giao Được @sanketpatil06 nhận Xem trên GitHub
models needs review
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

### Is your feature request related to a specific problem?
Yes. When running multi-agent systems locally using `Litert-lm`, concurrent agents compete simultaneously for limited local inference resources. Currently, the standard workaround is to stand up an external serving layer (like the `lit serve` method) and interact with it by reusing the standard `Gemini` or `LiteLlm` classes. This adds unnecessary overhead and removes fine-grained control over local resources.

### Describe the Solution You'd Like
A native `LitertModel` integration that extends the `BaseLLM` class specifically optimized for local orchestration. The two critical requirements are:
1. **Built-in Request Queuing:** A mechanism to queue requests at the model instantiation level so multiple local agents don't blast the local inference engine simultaneously and cause resource starvation.
2. **KV Cache Management:** Exposed controls to manage the KV cache specifically for instruction-based agents to optimize context switching and memory usage.

### Impact on your work
This is critical for building robust, offline multi-agent setups. It allows developers to run complex agent hierarchies locally without hitting OOM errors or having to maintain a separate, heavy serving infrastructure alongside ADK.

### Willingness to contribute
Yes. I already have a working implementation utilizing a custom `BaseLLM` subclass to handle the queueing and cache management, and I would be happy to submit a PR.

---
## 🟡 Recommended Information

**### Describe Alternatives You've Considered**
1. Using the `lit serve` method to create a local endpoint and pointing a standard ADK cloud model class at it. This works but adds unnecessary latency, architectural complexity, and abstracts away direct cache control.
2. Relying on the standard `LiteLlm` wrapper, which does not inherently solve the multi-agent concurrent queuing problem for constrained local resources.

### Proposed API / Implementation
```python
from google.adk.models import BaseLLM
import queue

class LiteRTModel(BaseLLM):
def __init__(self, model_path: str, manage_kv_cache: bool = True):
super().__init__()
# Internal queue to manage concurrent agent requests
self._request_queue = queue.Queue()
self.manage_kv_cache = manage_kv_cache
# ... Litert-lm initialization ...

# Implementation overriding standard generation to utilize the queue
# and handle context swapping via the KV cache for different instruction agents.
```

### Additional Context
This aligns with ADK's goal of being deployment-agnostic by making local-first, offline execution a first-class citizen for complex multi-agent workflows.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.