agentscope-ai / agentscope-ai/agentscope

[Bug]: context-compression LLM call's token usage is never recorded anywhere

Open
#2,151 4 comments 0 reactions 1 assignee Claimed by @nolanchic View on GitHub
Dominant language
Python
Stars
31.5k
Forks
3.5k
Avg merge
1d 23h
Merged PRs (30d)
95

Description

### Prerequisites

- [x] I have searched the existing issues and discussions, and this is not a duplicate.
- [x] This is a bug, not a usage question.

### Background / Description

When an agent's context grows past the compression threshold, AgentScope makes an internal LLM call to summarize/compress the conversation before continuing. That call spends real input/output tokens with the provider, same as any other model call.

Every other model call the agent makes (the normal reasoning/reply calls) ends up with its token usage recorded on the agent's state, so anything built on top of usage tracking (cost estimation, budget limits, per-turn `ModelCallEndEvent` listeners, etc.) sees it. The compression call is the one exception: its usage is never recorded anywhere I can find and never shows up in any event, so any application relying on AgentScope's usage numbers will systematically undercount cost/tokens by the (potentially large, since it summarizes the whole reserved-out context) compression call, with no error or warning.

### Error Messages

Not applicable: this is not a crash or exception. It's a silent gap: the operation completes successfully (the compression happens, the summary is produced and applied), but the token cost of doing so is dropped rather than recorded.

### Steps to Reproduce

Minimal repro using a stub model (no live API needed) that returns a `StructuredResponse` carrying non-zero usage, exactly like a real provider adapter does. It forces compression to trigger on the first check, then compares the usage recorded after compression against the usage recorded after an ordinary model reply with the identical payload:

```python
import asyncio
from pydantic import BaseModel

from agentscope.agent._agent import Agent
from agentscope.model._base import ChatModelBase
from agentscope.model._model_response import StructuredResponse
from agentscope.model._model_usage import ChatUsage
from agentscope.message import TextBlock

class DummyCredential:
pass

class Summary(BaseModel):
task_overview: str = "t"
current_state: str = "s"
important_discoveries: str = "d"
next_steps: str = "n"
context_to_preserve: str = "c"

class StubParameters(BaseModel):
pass

class StubModel(ChatModelBase):
def __init__(self):
super().__init__(
credential=DummyCredential(),
model="stub-model",
parameters=StubParameters(),
stream=False,
context_size=1000,
)
self.structured_calls = 0

async def count_tokens(self, *args, **kwargs) -> int:
return 900 # always over trigger_ratio * context_size

async def generate_structured_output(self, messages, structured_model, **kwargs):
self.structured_calls += 1
return StructuredResponse(
content=Summary().model_dump(),
usage=ChatUsage(input_tokens=12345, output_tokens=678, time=1.23),
)

async def __call__(self, *args, **kwargs):
raise NotImplementedError("not exercised in this repro")

async def main():
model = StubModel()
agent = Agent(name="tester", system_prompt="You are a helpful assistant.", model=model)

agent.state.append_context("user", [TextBlock(type="text", text="hello " * 50)])
agent.state.context[-1].usage = None

await agent._compress_context_impl()

print("structured_output calls:", model.structured_calls)
print("summary set:", bool(agent.state.summary))
print("usage after compression:", agent.state.context[-1].usage if agent.state.context else None)

# Control: an ordinary reply's usage IS recorded on the context tail.
agent._save_to_context(
[TextBlock(type="text", text="a normal reply")],
ChatUsage(input_tokens=12345, output_tokens=678, time=1.23),
)
print("usage after an ordinary reply (control):", agent.state.context[-1].usage)

asyncio.run(main())
```

Output on current `main`:

```
structured_output calls: 1
summary set: True
usage after compression: None
usage after an ordinary reply (control): input_tokens=12345 output_tokens=678
```

The compression call fired (`structured_calls == 1`) and completed successfully (`summary set: True`), but its 12345/678 token usage is nowhere afterward. The identical usage payload attached to an ordinary reply (control) is recorded correctly, so this isn't "usage tracking is absent," it's specific to the context-compression call.

### Environment

- AgentScope Version: 2.0.4.post1 (installed from `main` @ `77318e0`)
- Python Version: 3.12.13
- OS: Linux (Debian, `python:3.12-slim` container)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.