agentscope-ai / agentscope-ai/QwenPaw

perf: prefix cache instability from unsorted tool schemas and interleaved env_context fields

Abierto
#6,952 1 comentario 0 reacciones 1 asignado Reclamado por @rayrayraykk Ver en GitHub
Lenguaje dominante
Python
Estrellas
34.9k
Forks
3.1k
Merge medio
1 d 15 h
PR fusionados (30 d)
225

Descripción

## Problem

The LLM prefix (KV) cache misses unnecessarily on repeated turns due to two sources of
non-determinism in the prompt construction:

1. **Tool schema ordering** — `ToolRegistry.filter()` iterates `self._descs.values()` in dict
insertion order. This is stable within a process but not across restarts or plugin load-order
changes. Tool schemas are the largest prefix segment in the LLM request (~3,000-8,000 tokens).

2. **Env context field interleaving** — `build_env_context()` emits per-session dynamic fields
(session_id, date, working_dir) interleaved with static fields (About, GitHub, Docs, Important
hints). Every call changes the dynamic fields, shifting all subsequent tokens including the
static hint block that should be cache-stable.

Both issues cause the prefix cache to invalidate on segments that should be reusable across turns.

## Root Cause Analysis

Six potential prefix cache instability sources were identified:

| # | Source | Status | Impact |
|---|--------|--------|--------|
| 1 | Tool schema ordering (dict insertion order) | **Fixable** | High — ~3-8K tokens |
| 2 | env_context dynamic fields interleaved in static content | **Fixable** | Medium — ~150-400 tokens |
| 3 | Memory tool injection (post-toolkit) | Low — same manager in practice | Low |
| 4 | Coding mode project_dir | Low — worktree paths are rare | Low |
| 5 | Continuation summary shifts (scroll compression) | Intentional design | N/A |
| 6 | Tool result folding (`[scroll folded]` markers) | Intentional design | N/A |

Items 5 and 6 are documented architectural decisions in the Scroll memory system.

## Proposed Fix

1. **Sort tool schemas** by name in `ToolRegistry.filter()` — one line (`out.sort(key=lambda d: d.name)`)
2. **Split static/dynamic sections** in `build_env_context()` — static prefix (About/GitHub/Docs/Important) first, dynamic tail (OS/date/session_id/working_dir) last

Both changes are verified to be on the critical path from `AgentBuilder.build()` through to the LLM API request.

## Verification

- Tool schemas: `list_tools()` → `ToolRegistry.filter()` (sorted) → `PolicyGuardedTool` wrapping → `Toolkit` → `AgentScope.Agent` → `tools` param in LLM request
- Env context: `build_env_context()` → `EnvContextContributor` (priority=90, last contributor) → system prompt tail

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.