Azure-Samples / Azure-Samples/ai-model-start

DeepSeek-R1-0528: reasoning_tokens count always 0; reasoning item emission intermittent

Open
#12 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Bicep
Stars
3
Forks
4
PR merge metrics
No merged PRs in 30d

Description

### Summary

When calling `DeepSeek-R1-0528` via the Responses API on a Foundry project endpoint, the structured `reasoning` output item is now emitted with `summary[].text` populated (great!), but two server-side issues remain:

1. **`reasoning_tokens` is always 0** even when summary text is clearly present
2. **Reasoning item emission is intermittent** — roughly half of responses to the same prompt return only a `message` item, no `reasoning` item

This makes it hard to (a) build accurate token-accounting/billing logic and (b) write reliable demos or tests that depend on the reasoning surface.

### Reproduction

```python
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

endpoint = ""
client = OpenAI(
base_url=endpoint.rstrip("/") + "/openai/v1",
api_key=get_bearer_token_provider(DefaultAzureCredential(), "https://ai.azure.com/.default"),
)

for i in range(4):
r = client.responses.create(
model="DeepSeek-R1-0528",
input="What's heavier: a pound of feathers or a pound of bricks? Think step by step.",
max_output_tokens=2000,
)
types = [it.type for it in r.output]
rt = r.usage.output_tokens_details.reasoning_tokens
print(f"run {i+1}: items={types} reasoning_tokens={rt}")
```

Observed (4 runs, same prompt and parameters):

```
run 1: items=['message'] reasoning_tokens=0
run 2: items=['reasoning', 'message'] reasoning_tokens=0
run 3: items=['message'] reasoning_tokens=0
run 4: items=['message'] reasoning_tokens=0
```

When the `reasoning` item is present, `summary[0].text` contains a multi-paragraph chain-of-thought summary. But `reasoning_tokens` stays at 0 in every case.

### Expected behavior

- `reasoning_tokens` should reflect the number of tokens consumed by the model's reasoning trace (the way it does for OpenAI o-series models) — so billing/quota tracking is accurate
- The `reasoning` output item should be emitted consistently for the same prompt/parameters (or there should be a documented way to force it on, e.g. `include=["reasoning"]`)

### Impact

- Apps using the universal walk-output-items pattern can't reliably show the reasoning summary to users (it appears ~50% of the time)
- Apps tracking compute cost see `reasoning_tokens=0` and may significantly under-report DeepSeek usage
- Demos and tutorials (like https://github.com/Azure-Samples/ai-model-start) need to caveat the behavior in code comments

### Environment

- Foundry account in East US 2
- `DeepSeek-R1-0528` deployment, version 1
- `openai==2.36.0`, `azure-identity==1.x`
- Project endpoint, `/openai/v1` base URL, EntraID `DefaultAzureCredential` with `https://ai.azure.com/.default` scope

### Cross-reference

PR #11 surfaces reasoning summaries across all 5 language examples and handles both the present and absent cases gracefully, but this server-side fix would let the templates be deterministic and the token accounting accurate.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.