agentscope-ai / agentscope-ai/QwenPaw
Performance bottleneck in distributed self-hosted providers: cache miss amplification without session-sticky routing
- 主要言語
- Python
- スター
- 34.9k
- フォーク
- 3.1k
- 平均マージ
- 1日 15時間
- マージ済み PR(30日)
- 225
説明
## Problem Summary
In distributed self-hosted OpenAI-compatible deployments (for example vLLM/SGLang behind a load balancer), CoPaw currently allows request fan-out without session-stable routing guidance. This can create a **performance bottleneck**: prefix/prompt cache hit rate drops significantly when requests from one conversation are routed to different workers.
## Why This Is a Performance Issue
- **Cross-worker routing breaks prefix cache locality**: each worker builds its own KV/prefix cache state, so conversation turns landing on different nodes repeatedly miss warm cache.
- **Latency regression**: more prefill work is repeated instead of reused.
- **Throughput/cost regression**: redundant token prefill increases compute per request and reduces effective QPS.
## Additional Gap
Some providers/frameworks do not support automatic incremental prefix caching and require explicit cache breakpoints in prompts/context. If breakpoints are not placed carefully (at minimum including the end of system prompt), cacheability can degrade further.
## Scope Proposal (Self-hosted/Custom providers only)
1. Add explicit guidance and guardrails for distributed self-hosted OpenAI-compatible providers:
- enable prefix caching on inference backend (vLLM docs)
- keep session-stable routing (sticky/session affinity)
2. Keep CoPaw behavior **opt-in** and non-breaking:
- inject affinity header only when enabled
- preserve user-provided headers (do not overwrite)
3. Validate affinity-related config loaded from editable config files to avoid invalid header values causing runtime issues.
## Acceptance Criteria
- Self-hosted custom provider can enable session-affinity routing with stable value per `session_id`.
- User-defined affinity header value has priority over auto-injection.
- Invalid persisted header config is sanitized or safely rejected with clear errors/warnings.
- Documentation clearly explains vLLM prefix caching + sticky routing recommendations and expected performance impact.
## Reference
- vLLM prefix caching: https://docs.vllm.ai/en/stable/design/v1/prefix_caching.html
## Related PR
- https://github.com/agentscope-ai/CoPaw/pull/882
コントリビューションガイド
評価
この issue はまだ評価されていません。