Azure / Azure/azure-rest-api-specs
cognitiveservices Deployment real-time percentage usage
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 444
Description
https://learn.microsoft.com/en-us/python/api/azure-mgmt-cognitiveservices/azure.mgmt.cognitiveservices.models.deployment?view=azure-python
### **Is your feature request related to a problem? Please describe.**
I want to load-balance between different OpenAI deployments based on their real-time percentage usage.
### **Describe the solution you'd like**
As part of the [Deployment class](https://learn.microsoft.com/en-us/python/api/azure-mgmt-cognitiveservices/azure.mgmt.cognitiveservices.models.deployment?view=azure-python), or any of its underlying properties, it would be helpful to have not only the configured rate limits, but also the real-time usage (especially for TPM, not just RPM).
### **Describe alternatives you've considered**
I have looked into Azure Metrics, but I found two problems with this approach:
- the percentage usage metrics are only for Provisioned Managed Throughput, whereas I am looking for Standard TPM usage
- metrics are not available in real-time
### **Additional context**
sample code:
```
client = CognitiveServicesManagementClient(credential=DefaultAzureCredential(), subscription_id=sub_id)
deployments = client.deployments.list(resource_group_name="some_rg", account_name="some_acct")
print('Deployments:')
for d in deployments:
...
```
sample out:
```
Deployments:
================
gpt-4-turbo
================
{
"additional_properties": {},
"id": "...",
"name": "gpt-4-turbo",
"type": "Microsoft.CognitiveServices/accounts/deployments",
"sku": "{'additional_properties': {}, 'name': 'Standard', 'tier': None, 'size': None, 'family': None, 'capacity': 10}",
"system_data": "{'additional_properties': {}, 'created_by': '...', 'created_by_type': 'User', 'created_at': datetime.datetime(2024, 4, 10, 15, 19, 31, 6416, tzinfo=), 'last_modified_by': 'radu@gocascade.ai', 'last_modified_by_type': 'User', 'last_modified_at': datetime.datetime(2024, 4, 10, 15, 19, 31, 6416, tzinfo=)}",
"etag": "...",
"properties": "{'additional_properties': {}, 'provisioning_state': 'Succeeded', 'model': , 'scale_settings': None, 'capabilities': {'chatCompletion': 'true'}, 'rai_policy_name': 'Microsoft.Default', 'call_rate_limit': None, 'rate_limits': [, ], 'version_upgrade_option': 'OnceCurrentVersionExpired'}"
}
================
Properties:
{
"additional_properties": {},
"provisioning_state": "Succeeded",
"model": "{'additional_properties': {}, 'format': 'OpenAI', 'name': 'gpt-4', 'version': '1106-Preview', 'source': None, 'call_rate_limit': None}",
"scale_settings": null,
"capabilities": {
"chatCompletion": "true"
},
"rai_policy_name": "Microsoft.Default",
"call_rate_limit": null,
"rate_limits": [
"{'additional_properties': {}, 'key': 'request', 'renewal_period': 10.0, 'count': 10.0, 'min_count': None, 'dynamic_throttling_enabled': None, 'match_patterns': None}",
"{'additional_properties': {}, 'key': 'token', 'renewal_period': 60.0, 'count': 10000.0, 'min_count': None, 'dynamic_throttling_enabled': None, 'match_patterns': None}"
],
"version_upgrade_option": "OnceCurrentVersionExpired"
}
```
If there is an existing solution to this, please let me know.
Thank you!
Contributor guide
Research direction
Start with the Deployment model documentation and the CognitiveServicesManagementClient deployments.list example in the issue. Investigate whether the Azure Cognitive Services API exposes real-time Standard TPM usage, beyond the configured rate_limits and Azure Metrics mentioned. Done would require a defined API/spec change that makes this usage available to callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100