agentscope-ai / agentscope-ai/QwenPaw

Performance bottleneck in distributed self-hosted providers: cache miss amplification without session-sticky routing

オープン
#1,003 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
34.9k
フォーク
3.1k
平均マージ
1日 15時間
マージ済み PR(30日)
225

説明

## Problem Summary
In distributed self-hosted OpenAI-compatible deployments (for example vLLM/SGLang behind a load balancer), CoPaw currently allows request fan-out without session-stable routing guidance. This can create a **performance bottleneck**: prefix/prompt cache hit rate drops significantly when requests from one conversation are routed to different workers.

## Why This Is a Performance Issue
- **Cross-worker routing breaks prefix cache locality**: each worker builds its own KV/prefix cache state, so conversation turns landing on different nodes repeatedly miss warm cache.
- **Latency regression**: more prefill work is repeated instead of reused.
- **Throughput/cost regression**: redundant token prefill increases compute per request and reduces effective QPS.

## Additional Gap
Some providers/frameworks do not support automatic incremental prefix caching and require explicit cache breakpoints in prompts/context. If breakpoints are not placed carefully (at minimum including the end of system prompt), cacheability can degrade further.

## Scope Proposal (Self-hosted/Custom providers only)
1. Add explicit guidance and guardrails for distributed self-hosted OpenAI-compatible providers:
- enable prefix caching on inference backend (vLLM docs)
- keep session-stable routing (sticky/session affinity)
2. Keep CoPaw behavior **opt-in** and non-breaking:
- inject affinity header only when enabled
- preserve user-provided headers (do not overwrite)
3. Validate affinity-related config loaded from editable config files to avoid invalid header values causing runtime issues.

## Acceptance Criteria
- Self-hosted custom provider can enable session-affinity routing with stable value per `session_id`.
- User-defined affinity header value has priority over auto-injection.
- Invalid persisted header config is sanitized or safely rejected with clear errors/warnings.
- Documentation clearly explains vLLM prefix caching + sticky routing recommendations and expected performance impact.

## Reference
- vLLM prefix caching: https://docs.vllm.ai/en/stable/design/v1/prefix_caching.html

## Related PR
- https://github.com/agentscope-ai/CoPaw/pull/882

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。