Azure / Azure/azure-rest-api-specs

cognitiveservices Deployment real-time percentage usage

Open
#28,825 1 comment 0 reactions 0 assignees View on GitHub
customer-reported feature-request Mgmt needs-team-attention question Service Attention
Dominant language
TypeSpec
Stars
3.1k
Forks
5.9k
Avg merge
2d 22h
Merged PRs (30d)
444

Description

https://learn.microsoft.com/en-us/python/api/azure-mgmt-cognitiveservices/azure.mgmt.cognitiveservices.models.deployment?view=azure-python

### **Is your feature request related to a problem? Please describe.**
I want to load-balance between different OpenAI deployments based on their real-time percentage usage.

### **Describe the solution you'd like**
As part of the [Deployment class](https://learn.microsoft.com/en-us/python/api/azure-mgmt-cognitiveservices/azure.mgmt.cognitiveservices.models.deployment?view=azure-python), or any of its underlying properties, it would be helpful to have not only the configured rate limits, but also the real-time usage (especially for TPM, not just RPM).

### **Describe alternatives you've considered**
I have looked into Azure Metrics, but I found two problems with this approach:
- the percentage usage metrics are only for Provisioned Managed Throughput, whereas I am looking for Standard TPM usage
- metrics are not available in real-time

### **Additional context**
sample code:
```
client = CognitiveServicesManagementClient(credential=DefaultAzureCredential(), subscription_id=sub_id)
deployments = client.deployments.list(resource_group_name="some_rg", account_name="some_acct")
print('Deployments:')
for d in deployments:
...
```

sample out:
```
Deployments:
================
gpt-4-turbo
================
{
"additional_properties": {},
"id": "...",
"name": "gpt-4-turbo",
"type": "Microsoft.CognitiveServices/accounts/deployments",
"sku": "{'additional_properties': {}, 'name': 'Standard', 'tier': None, 'size': None, 'family': None, 'capacity': 10}",
"system_data": "{'additional_properties': {}, 'created_by': '...', 'created_by_type': 'User', 'created_at': datetime.datetime(2024, 4, 10, 15, 19, 31, 6416, tzinfo=), 'last_modified_by': 'radu@gocascade.ai', 'last_modified_by_type': 'User', 'last_modified_at': datetime.datetime(2024, 4, 10, 15, 19, 31, 6416, tzinfo=)}",
"etag": "...",
"properties": "{'additional_properties': {}, 'provisioning_state': 'Succeeded', 'model': , 'scale_settings': None, 'capabilities': {'chatCompletion': 'true'}, 'rai_policy_name': 'Microsoft.Default', 'call_rate_limit': None, 'rate_limits': [, ], 'version_upgrade_option': 'OnceCurrentVersionExpired'}"
}
================
Properties:
{
"additional_properties": {},
"provisioning_state": "Succeeded",
"model": "{'additional_properties': {}, 'format': 'OpenAI', 'name': 'gpt-4', 'version': '1106-Preview', 'source': None, 'call_rate_limit': None}",
"scale_settings": null,
"capabilities": {
"chatCompletion": "true"
},
"rai_policy_name": "Microsoft.Default",
"call_rate_limit": null,
"rate_limits": [
"{'additional_properties': {}, 'key': 'request', 'renewal_period': 10.0, 'count': 10.0, 'min_count': None, 'dynamic_throttling_enabled': None, 'match_patterns': None}",
"{'additional_properties': {}, 'key': 'token', 'renewal_period': 60.0, 'count': 10000.0, 'min_count': None, 'dynamic_throttling_enabled': None, 'match_patterns': None}"
],
"version_upgrade_option": "OnceCurrentVersionExpired"
}
```

If there is an existing solution to this, please let me know.

Thank you!

Contributor guide

Open the contributing guide

Research direction

Start with the Deployment model documentation and the CognitiveServicesManagementClient deployments.list example in the issue. Investigate whether the Azure Cognitive Services API exposes real-time Standard TPM usage, beyond the configured rate_limits and Azure Metrics mentioned. Done would require a defined API/spec change that makes this usage available to callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
api, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.