agentscope-ai / agentscope-ai/QwenPaw

perf: prefix cache instability from unsorted tool schemas and interleaved env_context fields

Aperta
#6,952 1 commento 0 reazioni 1 assegnatario Rivendicata da @rayrayraykk Vedi su GitHub
Lingua principale
TypeScript
Stelle
35k
Fork
3.1k
Merge medio
1g 13h
PR unite (30g)
228

Descrizione

## Problem

The LLM prefix (KV) cache misses unnecessarily on repeated turns due to two sources of
non-determinism in the prompt construction:

1. **Tool schema ordering** — `ToolRegistry.filter()` iterates `self._descs.values()` in dict
insertion order. This is stable within a process but not across restarts or plugin load-order
changes. Tool schemas are the largest prefix segment in the LLM request (~3,000-8,000 tokens).

2. **Env context field interleaving** — `build_env_context()` emits per-session dynamic fields
(session_id, date, working_dir) interleaved with static fields (About, GitHub, Docs, Important
hints). Every call changes the dynamic fields, shifting all subsequent tokens including the
static hint block that should be cache-stable.

Both issues cause the prefix cache to invalidate on segments that should be reusable across turns.

## Root Cause Analysis

Six potential prefix cache instability sources were identified:

| # | Source | Status | Impact |
|---|--------|--------|--------|
| 1 | Tool schema ordering (dict insertion order) | **Fixable** | High — ~3-8K tokens |
| 2 | env_context dynamic fields interleaved in static content | **Fixable** | Medium — ~150-400 tokens |
| 3 | Memory tool injection (post-toolkit) | Low — same manager in practice | Low |
| 4 | Coding mode project_dir | Low — worktree paths are rare | Low |
| 5 | Continuation summary shifts (scroll compression) | Intentional design | N/A |
| 6 | Tool result folding (`[scroll folded]` markers) | Intentional design | N/A |

Items 5 and 6 are documented architectural decisions in the Scroll memory system.

## Proposed Fix

1. **Sort tool schemas** by name in `ToolRegistry.filter()` — one line (`out.sort(key=lambda d: d.name)`)
2. **Split static/dynamic sections** in `build_env_context()` — static prefix (About/GitHub/Docs/Important) first, dynamic tail (OS/date/session_id/working_dir) last

Both changes are verified to be on the critical path from `AgentBuilder.build()` through to the LLM API request.

## Verification

- Tool schemas: `list_tools()` → `ToolRegistry.filter()` (sorted) → `PolicyGuardedTool` wrapping → `Toolkit` → `AgentScope.Agent` → `tools` param in LLM request
- Env context: `build_env_context()` → `EnvContextContributor` (priority=90, last contributor) → system prompt tail

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by tracing AgentBuilder.build() into ToolRegistry.filter() and build_env_context(), then follow the verification paths described in the issue to the LLM request. Done means tool schemas have deterministic ordering and static environment fields remain ahead of the dynamic tail, with prefix-stable content across repeated turns.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
performance
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
38/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.