[Azure monitor] ignore_unsupported does not skip HTTP 400 for unsupported metric namespaces
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Summary
`ignore_unsupported: true` should skip a resource when its namespace is not a supported Azure Monitor platform metric namespace. Today a HTTP 400 from the `metricDefinitions` API is still fatal. The `azure.monitor` metricset returns the error. The Elastic Agent unit changes from `HEALTHY` to `DEGRADED`. Collection stops for every resource in that unit, including types that are valid.
This is the Beats defect behind [elastic/integrations#20848](https://github.com/elastic/integrations/issues/20848).
## Version
- Elastic Stack 9.5.2
- Elastic Agent 9.4.4
- Azure Resource Metrics integration 1.14.0
- Current `main` still has the same path in `x-pack/metricbeat/module/azure`
## Operating System
Linux (AKS). The failure is in the Azure Monitor API client, not in the OS.
## Expected behaviour
When `ignore_unsupported` is true and Azure returns HTTP 400 because the namespace is not a supported platform metric namespace:
1. Metricbeat logs an info or warning line.
2. Metricbeat skips that resource.
3. Metricbeat continues collection for the other resources in the same cycle.
4. The unit stays `HEALTHY`.
That is the documented purpose of `ignore_unsupported` on `MetricConfig`.
## Actual behaviour
`GetMetricDefinitionsWithRetry` returns the HTTP 400. `mapMetrics` returns that error before it reads `IgnoreUnsupported`. `InitResources` then fails the whole refresh. The Agent unit becomes `DEGRADED`.
`ignore_unsupported` only applies after a successful definitions call that returns an empty list. It does not apply to a 400 from Azure.
Example error:
```
Error fetching data for metricset azure.monitor: GET https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.Insights/activityLogAlerts//providers/Microsoft.Insights/metricDefinitions
RESPONSE 400: 400 Bad Request
ERROR CODE: BadRequest
{
"code": "BadRequest",
"message": "Microsoft.Insights/activityLogAlerts is not a supported platform metric namespace, ..."
}
```
Unit state: `HEALTHY` to `DEGRADED`.
## Affected code
### 1. HTTP 400 is fatal before `ignore_unsupported` is checked
`GetMetricDefinitionsWithRetry` returns on error before `IgnoreUnsupported` is checked:
- Sequential:
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor/client_helper.go#L39-L42
- Concurrent:
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor/client_helper_concurrent.go#L49-L52
`ignore_unsupported` is only used later, when the call succeeds and the list is empty:
- Sequential:
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor/client_helper.go#L46-L53
- Concurrent:
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor/client_helper_concurrent.go#L56-L61
### 2. Non-429 errors are always returned
`GetMetricDefinitionsWithRetry` and `sleepIfPossible` retry HTTP 429 only. HTTP 400 is returned as an error.
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor_service.go#L227-L228
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/monitor_service.go#L265-L268
### 3. One mapping error fails the whole unit
`InitResources` returns on the first `mapMetrics` error. Remaining resource configs are not processed.
https://github.com/elastic/beats/blob/99112f7e4dd543d01405d0461721a92d8960f8b9/x-pack/metricbeat/module/azure/client.go#L119-L121
Contributor guide
Research direction
Start with GetMetricDefinitionsWithRetry in x-pack/metricbeat/module/azure/monitor/client_helper.go and client_helper_concurrent.go, then trace mapMetrics and InitResources in client.go. Check the retry and error paths in monitor_service.go; done means an unsupported-namespace HTTP 400 is skipped when ignore_unsupported is true, other resources continue, and the unit remains HEALTHY.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100