elastic / elastic/integrations
[Azure] Throttling when too many requests are sent to Azure Billing API
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Azure Billing Metrics [azure_billing]
### Dataset Name
azure.billing
### Integration Version
1.8.0
### Agent Version
8.18.2
### Agent Output Type
elasticsearch
### Elasticsearch Version
8.18.2
### OS Version and Architecture
Kubernetes
### Software/API Version
_No response_
### Error Message
```
- unit_id: azure/metrics-default-azure/metrics-billing-xxxx
unit_type: 0
state: 3
message: |
Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/subscriptions/yyyy/providers/Microsoft.Consumption/usageDetails
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Too many requests. Please retry after 60 seconds."
}
}
--------------------------------------------------------------------------------
payload:
streams:
azure/metrics-azure.billing-xxxx
error: |
Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/subscriptions/yyyy/providers/Microsoft.Consumption/usageDetails
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Too many requests. Please retry after 60 seconds."
}
}
--------------------------------------------------------------------------------
status: DEGRADED
```
### Event Original
_No response_
### What did you do?
I have deployed one Elastic agent with several azure billing integrations (one for each of my subscriptions)
It works well for one or two billing integrations, but does not work when I add up some subscriptions (about 20 in my current use case)
### What did you see?
I've got the 429 error...
### What did you expect to see?
Billing data on Elasticsearch
I expect the agent/integrations to handle the 429 and to retry accordingly (something like what was handled on https://github.com/elastic/integrations/issues/8575)
### Anything else?
💪
Contributor guide
Assessment
This issue has not been assessed yet.