agentscope-ai / agentscope-ai/agentscope

[Bug]: Context compaction fails and crashes with 400 error due to token underestimation in count_tokens

Đang mở
#2,385 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
31.5k
Fork
3.5k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
95

Mô tả

### Prerequisites

- [x] I have searched the existing [issues](https://github.com/agentscope-ai/agentscope/issues) and [discussions](https://github.com/agentscope-ai/agentscope/discussions), and this is not a duplicate.
- [x] This is a bug, not a usage question. (For questions, please use [Discussions](https://github.com/agentscope-ai/agentscope/discussions/new?category=general) instead.)

### Background / Description

In multi-turn agent sessions involving non-ASCII text (e.g., Chinese), numerical data (financial records, SQL), or tool outputs (JSON), AgentScope's context compaction and length guardrails fail completely. This causes requests to exceed the backend model's context window and crash with **HTTP 400 (`context_length_exceeded` / `BadRequestError`)**.

### Root Cause
`ChatModelBase.count_tokens` uses a fixed heuristic (`len(utf8_bytes) / 4`). While this works for standard English prose, it systematically underestimates tokens by **2x to 3x** on non-ASCII text, numbers, and structured tool outputs.

Because of this underestimation:
1. The session is mistakenly estimated to be well below the `0.8 * context_window` compaction threshold, bypassing compaction.
2. The full context is sent to the LLM backend (e.g., vLLM / OpenAI-compatible endpoint), which rejects it with HTTP 400.
3. The internal `drop-oldest` retry mechanism inside memory compression also relies on `count_tokens`, so it fails to detect the overflow, terminating the entire agent task.

### Benchmark Data (Tested against Qwen on vLLM)
We compared `ChatModelBase.count_tokens` against the ground-truth `usage.input_tokens` returned by the model backend:

| Workload Scenario | Chars | UTF-8 Bytes | AgentScope Estimate (`bytes/4`) | Actual Usage (`input_tokens`) | Underestimation Ratio |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Pure English query | 95 | 95 | 24 | 66 | **+175.0% (2.75x)** |
| Pure Chinese query | 44 | 124 | 31 | 74 | **+138.7% (2.39x)** |
| Financial & SQL Data | 377 | 473 | 118 | 315 | **+166.9% (2.67x)** |
| Tool Result (JSON) | 344 | 344 | 86 | 266 | **+209.3% (3.09x)** |
| Multi-turn Mix Context | 531 | 587 | 147 | 271 | **+84.4% (1.84x)** |

*(Note: Both streaming final chunk and non-streaming responses reliably return `ChatUsage(input_tokens, output_tokens)`).*

### Error Messages

```shell
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 65536 tokens. However, your messages resulted in 65537 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 400}}

Traceback (most recent call last):
File "agentscope/agent/_react_agent.py", line 150, in reply
response = await self._model(messages)
File "agentscope/model/_openai_model.py", line 120, in __call__
return await self._async_client.chat.completions.create(...)
openai.BadRequestError: Error code: 400 - {'message': 'This model maximum context length is 65536 tokens...'}
```

### Steps to Reproduce

1. Code:

```python
import asyncio
from agentscope.model import OpenAIChatModel, OpenAICredential
from agentscope.message import Msg, TextBlock

async def main():
model = OpenAIChatModel(
model_name="Qwen3.8-27B",
credential=OpenAICredential(api_key="your_key", base_url="http://localhost:8000/v1"),
stream=False,
)

# Financial SQL query with dense numbers
text = (
"SELECT order_id, sum(amount * (1 - discount)) as net_total "
"FROM orders WHERE amount > 10000.50 AND customer_id = 88392019482 "
"GROUP BY order_id HAVING net_total > 500000.00;"
)
msg = Msg(name="user", role="user", content=[TextBlock(type="text", text=text)])

# 1. AgentScope Estimate
estimated = await model.count_tokens([msg], None)
# 2. Actual LLM Provider Usage
resp = await model([msg])
actual = resp.usage.input_tokens

print(f"AgentScope count_tokens: {estimated}")
print(f"Model Actual input_tokens: {actual}")
print(f"Underestimated ratio: {actual / estimated:.2f}x")

if __name__ == "__main__":
asyncio.run(main())
```
2. Run: python reproduce.py
3. See output: count_tokens reports ~118 tokens while actual LLM usage is ~315 tokens (underestimated by ~2.67x). In long multi-turn sessions, this error compounds until physical context limit is exceeded and the backend returns 400.

### Environment

```text
- OS: Linux (Ubuntu 22.04 / Docker)
- Python Version: 3.11.15
- AgentScope Version: 2.0.4 / 2.0.6
- Backend Model: Qwen3.8-27B (hosted via vLLM / OpenAI-compatible endpoint)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.