[Question/Performance] Full message history is re-tokenized on every coding-agent turn

Background

The current coding-agent adapters receive OpenAI/Anthropic messages from
Claude Code or Codex and render the full message history on every turn:

https://github.com/THUDM/slime/blob/main/slime/agent/adapters/common.py

translated, tools_schema = self._translate(body)
prompt_ids = _render_token_ids(
    translated,
    tokenizer,
    tools=tools_schema,
    add_generation_prompt=True,
)

_render_token_ids() calls tokenizer.apply_chat_template(..., tokenize=True)
for the complete message list.

I understand that the exact output_ids and rollout logprobs returned by
SGLang are preserved by TrajectoryManager, and that token drift is handled by
realign/fork/masking. Therefore this question is about CPU scalability rather
than token provenance correctness.

Concern

For a long-horizon agent trajectory, every request normally contains the
complete and growing message history.

If the prompt lengths over N turns are:

L1, L2, ..., LN

the adapter performs approximately:

L1 + L2 + ... + LN

tokens of CPU tokenization work.

When the context grows approximately linearly with the number of turns, this
becomes close to O(N^2) cumulative tokenization work per trajectory.

This may become significant for:

• 128K to 1M-token coding-agent trajectories;

• many concurrent agent sessions sharing one adapter;

• tool observations containing large source files or compiler logs;

• repeated tool schemas and system prompts;

• SGLang prefix-cache hits that make GPU prefill cheaper while CPU still
  re-renders and re-tokenizes the full prompt;

• auto-compaction, sub-agent and retry paths.

The tokenization call is also synchronous in the aiohttp request path. A long
tokenization operation may therefore block other sessions served by the same
adapter event loop.

Questions

1. Is full-history re-tokenization on every coding-agent turn the intended
   design?

2. Has its CPU time, event-loop blocking time or throughput been benchmarked
   for long contexts and concurrent sessions?

3. Are there existing metrics for:

   • chat-template rendering latency;
   • tokenizer latency;
   • number of input tokens re-tokenized per turn;
   • adapter queueing latency;
   • tokenizer CPU utilization?

4. Is there a plan to support session-scoped incremental tokenization or an
   append-only token checkpoint, similar to a session-server TITO path?

5. Was the removed radix-tree middleware from #242/#1735 considered for the
   coding-agent adapter, or was it removed because its consistency, memory or
   maintenance cost was too high?

Possible directions

Some possible approaches, with full rendering retained as a correctness
fallback:

1. Keep a per-session token checkpoint and normalized message prefix.

2. When the next request is append-only and the tokenizer/chat-template/tools
   configuration is unchanged, tokenize only the appended messages and merge
   them with the checkpoint.

3. Validate the incremental result against template-specific invariants and
   fall back to full rendering for retries, compaction, rewrites, tool-schema
   changes or ambiguous token boundaries.

4. Move unavoidable full tokenization to a bounded worker pool so it does not
   block the adapter event loop.

5. Add latency and processed-token metrics before selecting an optimization.

A naive concatenation of independently tokenized message fragments may be
incorrect because of tokenizer boundary behavior and chat-template suffixes,
so this likely requires a model/template-aware implementation rather than
simply caching tokenizer(message) results.

Related work

• #117: token-in/token-out proposal
• #242: radix-tree based multi-turn TITO
• #1735: removal of the radix-tree middleware
• #1359: native TITO agent loop through strands-sglang
• #2005: current coding-agent TrajectoryManager and retokenization-drift handling

Could the maintainers clarify whether this CPU cost is already considered
acceptable, has been measured, or is planned to be optimized?

What I've Tried

• I have read the relative dissusion and docs, but not find any performence issues

Environment (if relevant)

• slime version:
• Python version:
• PyTorch version:
• CUDA/ROCm version:
• GPU type and count:
• OS:

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Start in slime/agent/adapters/common.py at _translate() and _render_token_ids(), then review the related work in issues #117, #242, #1735, #1359, and #2005. Measure full-history tokenization and event-loop impact for long, concurrent coding-agent sessions, and use those results to clarify whether incremental tokenization, worker isolation, or existing behavior is the intended direction.