agentscope-ai / agentscope-ai/QwenPaw
perf: prefix cache instability from unsorted tool schemas and interleaved env_context fields
- Lingua principale
- TypeScript
- Stelle
- 35k
- Fork
- 3.1k
- Merge medio
- 1g 13h
- PR unite (30g)
- 228
Descrizione
## Problem
The LLM prefix (KV) cache misses unnecessarily on repeated turns due to two sources of
non-determinism in the prompt construction:
1. **Tool schema ordering** — `ToolRegistry.filter()` iterates `self._descs.values()` in dict
insertion order. This is stable within a process but not across restarts or plugin load-order
changes. Tool schemas are the largest prefix segment in the LLM request (~3,000-8,000 tokens).
2. **Env context field interleaving** — `build_env_context()` emits per-session dynamic fields
(session_id, date, working_dir) interleaved with static fields (About, GitHub, Docs, Important
hints). Every call changes the dynamic fields, shifting all subsequent tokens including the
static hint block that should be cache-stable.
Both issues cause the prefix cache to invalidate on segments that should be reusable across turns.
## Root Cause Analysis
Six potential prefix cache instability sources were identified:
| # | Source | Status | Impact |
|---|--------|--------|--------|
| 1 | Tool schema ordering (dict insertion order) | **Fixable** | High — ~3-8K tokens |
| 2 | env_context dynamic fields interleaved in static content | **Fixable** | Medium — ~150-400 tokens |
| 3 | Memory tool injection (post-toolkit) | Low — same manager in practice | Low |
| 4 | Coding mode project_dir | Low — worktree paths are rare | Low |
| 5 | Continuation summary shifts (scroll compression) | Intentional design | N/A |
| 6 | Tool result folding (`[scroll folded]` markers) | Intentional design | N/A |
Items 5 and 6 are documented architectural decisions in the Scroll memory system.
## Proposed Fix
1. **Sort tool schemas** by name in `ToolRegistry.filter()` — one line (`out.sort(key=lambda d: d.name)`)
2. **Split static/dynamic sections** in `build_env_context()` — static prefix (About/GitHub/Docs/Important) first, dynamic tail (OS/date/session_id/working_dir) last
Both changes are verified to be on the critical path from `AgentBuilder.build()` through to the LLM API request.
## Verification
- Tool schemas: `list_tools()` → `ToolRegistry.filter()` (sorted) → `PolicyGuardedTool` wrapping → `Toolkit` → `AgentScope.Agent` → `tools` param in LLM request
- Env context: `build_env_context()` → `EnvContextContributor` (priority=90, last contributor) → system prompt tail
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start by tracing AgentBuilder.build() into ToolRegistry.filter() and build_env_context(), then follow the verification paths described in the issue to the LLM request. Done means tool schemas have deterministic ordering and static environment fields remain ahead of the dynamic tail, with prefix-stable content across repeated turns.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- performance
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 38/100