elastic / elastic/integrations

[Azure] Throttling when too many requests are sent to Azure Billing API

Open
#14,313 2 comments 0 reactions 0 assignees View on GitHub
Integration:azure_billing needs:triage Team:Obs-InfraObs
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Azure Billing Metrics [azure_billing]

### Dataset Name

azure.billing

### Integration Version

1.8.0

### Agent Version

8.18.2

### Agent Output Type

elasticsearch

### Elasticsearch Version

8.18.2

### OS Version and Architecture

Kubernetes

### Software/API Version

_No response_

### Error Message

```
- unit_id: azure/metrics-default-azure/metrics-billing-xxxx
unit_type: 0
state: 3
message: |
Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/subscriptions/yyyy/providers/Microsoft.Consumption/usageDetails
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Too many requests. Please retry after 60 seconds."
}
}
--------------------------------------------------------------------------------
payload:
streams:
azure/metrics-azure.billing-xxxx
error: |
Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/subscriptions/yyyy/providers/Microsoft.Consumption/usageDetails
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Too many requests. Please retry after 60 seconds."
}
}
--------------------------------------------------------------------------------
status: DEGRADED
```

### Event Original

_No response_

### What did you do?

I have deployed one Elastic agent with several azure billing integrations (one for each of my subscriptions)

It works well for one or two billing integrations, but does not work when I add up some subscriptions (about 20 in my current use case)

### What did you see?

I've got the 429 error...

### What did you expect to see?

Billing data on Elasticsearch

I expect the agent/integrations to handle the 429 and to retry accordingly (something like what was handled on https://github.com/elastic/integrations/issues/8575)

### Anything else?

💪

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.