Azure / Azure/azure-sdk-for-python

Return Token Count

Open
#41,241 2 comments 1 reaction 1 assignee Claimed by @singankit View on GitHub
customer-reported Evaluation feature-request needs-team-attention Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

## Missing Functionality
When running queries to Azure OpenAI, the api always returns the number of tokens consumed (input tokens + output tokens) as shown below. This information is currently not returned by the evaluation package.

```json
{
"body": {
....
"choices": [
{
"text": "es\n\nWhat do you call a mango who's in charge?\n\nThe head mango.",
"index": 0,
"finish_reason": "stop",
"logprobs": null
}
],
"usage": { ## !!! This is what I need !!!
"completion_tokens": 20,
"prompt_tokens": 6,
"total_tokens": 26
}
}
}
```

## Why is this necessary ?
To ensure latency for solution in production, customers need to estimate the PTUs required taking into account not only the number of requests but also how much the evaluation process will consume.
Currently to estimate the total number of tokens, the only way is to manually load the prompt files for the different metrics (relevancy, roundedness) and use tiktoken to estimate the number of tokens. The same needs to be done for the answer and reasoning in the evaluation's response.

## Expected Behaviour
In the response of each evaluation request, I'd expect an additional field ("usage" like the AOAI's case) that lists the number of token used.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.