agentscope-ai / agentscope-ai/QwenPaw
perf: prefix cache instability from unsorted tool schemas and interleaved env_context fields
- Lenguaje dominante
- Python
- Estrellas
- 34.9k
- Forks
- 3.1k
- Merge medio
- 1 d 15 h
- PR fusionados (30 d)
- 225
Descripción
## Problem
The LLM prefix (KV) cache misses unnecessarily on repeated turns due to two sources of
non-determinism in the prompt construction:
1. **Tool schema ordering** — `ToolRegistry.filter()` iterates `self._descs.values()` in dict
insertion order. This is stable within a process but not across restarts or plugin load-order
changes. Tool schemas are the largest prefix segment in the LLM request (~3,000-8,000 tokens).
2. **Env context field interleaving** — `build_env_context()` emits per-session dynamic fields
(session_id, date, working_dir) interleaved with static fields (About, GitHub, Docs, Important
hints). Every call changes the dynamic fields, shifting all subsequent tokens including the
static hint block that should be cache-stable.
Both issues cause the prefix cache to invalidate on segments that should be reusable across turns.
## Root Cause Analysis
Six potential prefix cache instability sources were identified:
| # | Source | Status | Impact |
|---|--------|--------|--------|
| 1 | Tool schema ordering (dict insertion order) | **Fixable** | High — ~3-8K tokens |
| 2 | env_context dynamic fields interleaved in static content | **Fixable** | Medium — ~150-400 tokens |
| 3 | Memory tool injection (post-toolkit) | Low — same manager in practice | Low |
| 4 | Coding mode project_dir | Low — worktree paths are rare | Low |
| 5 | Continuation summary shifts (scroll compression) | Intentional design | N/A |
| 6 | Tool result folding (`[scroll folded]` markers) | Intentional design | N/A |
Items 5 and 6 are documented architectural decisions in the Scroll memory system.
## Proposed Fix
1. **Sort tool schemas** by name in `ToolRegistry.filter()` — one line (`out.sort(key=lambda d: d.name)`)
2. **Split static/dynamic sections** in `build_env_context()` — static prefix (About/GitHub/Docs/Important) first, dynamic tail (OS/date/session_id/working_dir) last
Both changes are verified to be on the critical path from `AgentBuilder.build()` through to the LLM API request.
## Verification
- Tool schemas: `list_tools()` → `ToolRegistry.filter()` (sorted) → `PolicyGuardedTool` wrapping → `Toolkit` → `AgentScope.Agent` → `tools` param in LLM request
- Env context: `build_env_context()` → `EnvContextContributor` (priority=90, last contributor) → system prompt tail
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.