Azure / Azure/azure-sdk-for-python

No usage tokens in Deep Research tool output

Open
#43,910 1 comment 0 reactions 0 assignees View on GitHub
AI Projects customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

The problem is the following: **The run's steps returned once the Deep Research job is done contain (almost) NO useful information regarding token consumption of o3-deep-research model nor BING calls.**

When one implements the deep research agent using the SDK as it is explained in the tutorial:
https://learn.microsoft.com/en-gb/azure/ai-foundry/agents/how-to/tools/deep-research-samples?pivots=python

and ones wants to get details regarding the tools usage (like BING calls or o3-deep-research token usage, not just the gpt-4o ones) one can simple do:

`run_steps = ai_project_client.agents.run_steps.list(thread_id=thread_id, run_id=run_id)`

and inspect each step in run_steps, but **NONE of those steps contain BING calls neither token usage for the o3-deep_research model**

The response instead contains a bunch of steps with ZERO token usage for message_creation type (gpt-4o), like this

`"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0,
"prompt_token_details": {
"cached_tokens": 0
}
}`

and one last step which contains 1 call tool to deep_research and some token consumption, something that looks like this:

`"step_details": {
"type": "tool_calls",
"tool_calls": [
{
"id": "call_SOME_ID",
"type": "deep_research",
"deep_research": {
"input": "{\"prompt\": \"SOME PROMPT\"}",
"output": null
}
}
]
},
"usage": {
"prompt_tokens": 4055,
"completion_tokens": 268,
"total_tokens": 4323,
"prompt_token_details": {
"cached_tokens": 0
}
}`

and after checking the Azure portal, I could confirmed that those tokens **ARE NOT** for the o3-deep-research model, but exactly the ones for gpt-4o.

**Expected Behaviour**: I would expect to get in those steps some with Non zero usage for gpt-4o calls and also some having BING calls (and of course the real token values usage for o3-deep-research model, which I can see in the UI and are definitely not the ones appearing in the corresponding step)

PD: Just for the record, inspecting those steps **does work**** in other examples, like for instance this one
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-agents/samples/agents_tools/sample_agents_bing_grounding.py

which makes me believe there is a bug for the Deep Research module

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.