Azure / Azure/azure-sdk-for-python

Return Token Count

Aperta
#41,241 2 commenti 1 reazione 1 assegnatario Rivendicata da @singankit Vedi su GitHub
customer-reported Evaluation feature-request needs-team-attention Service Attention
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g 2h
PR unite (30g)
202

Descrizione

## Missing Functionality
When running queries to Azure OpenAI, the api always returns the number of tokens consumed (input tokens + output tokens) as shown below. This information is currently not returned by the evaluation package.

```json
{
"body": {
....
"choices": [
{
"text": "es\n\nWhat do you call a mango who's in charge?\n\nThe head mango.",
"index": 0,
"finish_reason": "stop",
"logprobs": null
}
],
"usage": { ## !!! This is what I need !!!
"completion_tokens": 20,
"prompt_tokens": 6,
"total_tokens": 26
}
}
}
```

## Why is this necessary ?
To ensure latency for solution in production, customers need to estimate the PTUs required taking into account not only the number of requests but also how much the evaluation process will consume.
Currently to estimate the total number of tokens, the only way is to manually load the prompt files for the different metrics (relevancy, roundedness) and use tiktoken to estimate the number of tokens. The same needs to be done for the answer and reasoning in the evaluation's response.

## Expected Behaviour
In the response of each evaluation request, I'd expect an additional field ("usage" like the AOAI's case) that lists the number of token used.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.