agentscope-ai / agentscope-ai/agentscope-java

[Bug] DashScope native API: cache_control placed at message level instead of content block level

Aperta
#1,363 0 commenti 0 reazioni 1 assegnatario Rivendicata da @flystar32 Vedi su GitHub
area/core/model bug
Lingua principale
Java
Stelle
5.6k
Fork
1.3k
Merge medio
4g 12h
PR unite (30g)
77

Descrizione

## Description

The explicit cache control implementation in both `DashScopeChatModel` and `OpenAIChatModel` places `cache_control` at the **message level**, but the [DashScope official documentation](https://help.aliyun.com/zh/model-studio/context-cache) requires it to be placed **inside the content block** (within the `content` array). This applies to both the DashScope native protocol and the OpenAI-compatible protocol.

## Current Behavior

When `GenerateOptions.cacheControl(true)` is enabled, both `DashScopeChatFormatter.applyCacheControl()` and `OpenAIBaseFormatter.applyCacheControl()` set `cache_control` on the message object directly (message level).

**DashScope formatter:**

```java
public void applyCacheControl(List messages) {
for (DashScopeMessage msg : messages) {
if ("system".equals(msg.getRole()) && msg.getCacheControl() == null) {
msg.setCacheControl(EPHEMERAL_CACHE_CONTROL); // message level
}
}
DashScopeMessage lastMsg = messages.get(messages.size() - 1);
if (lastMsg.getCacheControl() == null) {
lastMsg.setCacheControl(EPHEMERAL_CACHE_CONTROL); // message level
}
}
```

**OpenAI formatter** has the same logic in `OpenAIBaseFormatter.applyCacheControl()`.

This produces the following JSON for both protocols:

```json
{
"role": "system",
"content": "You are a helpful assistant.",
"cache_control": {"type": "ephemeral"}
}
```

## Expected Behavior

Per the [official documentation](https://help.aliyun.com/zh/model-studio/context-cache), `cache_control` must be placed **inside a content block**, and `content` must be in **array format**:

```json
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant.",
"cache_control": {"type": "ephemeral"}
}
]
}
```

The documentation states:

> 需将 content 字段改为数组形式,并添加 cache_control 字段

This format requirement applies to both OpenAI-compatible and DashScope native protocols when calling DashScope models.

## Issues Identified

1. **`cache_control` is placed at the wrong level** — should be inside content blocks, not at message level. This affects both `DashScopeChatFormatter` and `OpenAIBaseFormatter`.
2. **Content part DTOs lack a `cache_control` field** — both `DashScopeContentPart` and `OpenAIMessage` content parts have no way to carry `cache_control` at the content block level.
3. **Multimodal messages are not handled** — when content is already a list of content parts, the `cache_control` still goes to message level and won't be recognized by the API.
4. **No guard for the 4-marker limit** — the documentation states a maximum of 4 `cache_control` markers per request. If there are multiple system messages (e.g., injected by SkillHook or LongTermMemoryHook), the limit may be exceeded silently.

## Suggested Fix

### DashScope native protocol

1. Add a `cache_control` field (`Map`) to `DashScopeContentPart`.
2. Modify `DashScopeChatFormatter.applyCacheControl()` to:
- Convert string `content` to array format (`List`) for target messages.
- Set `cache_control` on the **last content block** within each target message.
3. Apply the same fix to `DashScopeMultiAgentFormatter.applyCacheControl()`.

### OpenAI-compatible protocol

1. Modify `OpenAIBaseFormatter.applyCacheControl()` with the same content-block-level approach.
2. Add `cache_control` support to OpenAI content part DTOs.

### Common

1. Add a guard to ensure no more than 4 `cache_control` markers per request.
2. Keep the existing message-level `cacheControl` fields for backward compatibility (manual metadata marking).

## Affected Classes

- `DashScopeChatFormatter`
- `DashScopeMultiAgentFormatter`
- `DashScopeContentPart`
- `DashScopeMessage`
- `OpenAIBaseFormatter`
- `OpenAIMessage`

## Discussion: Should `applyCacheControl()` auto-mark messages?

The current `applyCacheControl()` strategy is: **"all system messages + last message"**. Given DashScope's prefix-matching caching mechanism, the strategy pattern itself is sound in theory:

- **Marking system messages** creates layered prefix cache blocks (A, AB, ABC…). Even if later messages change, the shorter prefix (e.g., just the stable system prompt) can still be hit — a reasonable tiered caching approach.
- **Marking the last message** caches the entire messages array as a complete prefix, which aligns with the official "continuous multi-turn dialog" pattern.

However, there are two practical concerns:

### 1. The 4-marker limit

The API enforces a hard limit of **4 `cache_control` markers per request**. If more than 4 markers are present, only the last 4 take effect. In AgentScope, multiple hooks can dynamically inject system messages (e.g., SkillHook, LongTermMemoryHook, RAGHook), making the number of system messages unpredictable at the formatter level. When the total marker count exceeds 4, the earliest system messages — which are typically the most stable and most valuable to cache — will lose their markers and fall out of the cache.

### 2. Dynamic content defeats prefix caching

The framework cannot distinguish between **stable** system messages (e.g., the user's own system prompt) and **dynamic** ones (e.g., RAG-retrieved knowledge, long-term memory summaries). In AgentScope's hook architecture, hooks like `GenericRAGHook` and `StaticLongTermMemoryHook` inject system messages whose **content changes on every request**. Marking these with `cache_control` means each request creates a new cache block (at 125% of standard input cost) that will likely **never be hit** — the prefix changes every time.

Only the user knows which parts of their messages are stable and worth caching. A blanket "mark all system messages" strategy applied at the formatter level cannot make this distinction.

### Suggestion

Consider making cache control **user-driven** rather than automatic:

- The existing `MessageMetadataKeys.CACHE_CONTROL` mechanism already allows users to mark individual `Msg` objects for caching via metadata, which flows through `applyCacheControlFromMetadata()`.
- The automatic `applyCacheControl()` strategy could be **removed or made opt-in**, letting users who understand their caching needs and cost tolerance decide which messages to mark.
- If keeping an automatic strategy, add a guard to enforce the 4-marker limit and prioritize stable prefixes (first system message + last message).

## References

- DashScope Context Cache documentation: https://help.aliyun.com/zh/model-studio/context-cache
- DashScope Java SDK correct usage: `MessageContentText.builder().text(...).cacheControl(CacheControl.builder().type("ephemeral").build()).build()`

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.