add_cloud_metadata: flaky behavior for azure provider reproduced on Azure VMs after VM restart
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
As per user report, we have been able to reproduce the fact the `add_cloud_metadata` is not always attaching the Azure metadata when running on a Azure VM, especially after a VM restart
Restarting Elastic Agent can randomly make it re-add the metadata again.
The `add_cloud_metadata` are implemented without any retry, so if the first attempt fails (e.g. while the VM is still lacking connectivity/access to network), the metadata will not be attached.
Still, in our tests we verified that changing the Elastic Agent Systemd unit to REQUIRE network as follows didn't help.
```
[Unit]
Description=Elastic Agent is a unified agent to observe, monitor and protect your system.
ConditionFileIsExecutable=/usr/bin/elastic-agent
After=network-online.target
Wants=network-online.target
```
The fact the processor is not attaching metadata can be seen in the EA diagnostic, under `global_processors.txt`:
```
components/beat-metrics-monitoring/global_processors.txt:add_cloud_metadata={}
components/filestream-monitoring/global_processors.txt:add_cloud_metadata={}
components/http-metrics-monitoring/global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
components/log-default/global_processors.txt:add_cloud_metadata={}
components/system-metrics-default/global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
```
Once EA is restarted, it might get all the components with the:
```
components/...all of them... /global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
```
This has been observed on Beats & Elastic Agent 8.19.x
### What to expect as a fix
The `add_cloud_metadata` processor should not be flaky after a VM restart. The resulting data produced with this provider could prevent security or observability alerts/detection rules to fail as some crucial metadata is missing.
### Possible reason?
Looking at the code, it seems we register several providers (based on config or otherwise we go to [defaults](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers.go#L58)) and then we do a [parallel probing](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers.go#L180).
I just noticed `azure` provider is only part of the non-fips build [here](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers_nofips.go#L27).
In 8.18, `azure` was statically initialized https://github.com/elastic/beats/blob/v8.18.0/libbeat/processors/add_cloud_metadata/providers.go
On 8.19, via https://github.com/elastic/beats/commit/0ea1263cf1d63509e53516daffdc5737b63b7c09 (due to FIPS builds), it became conditionally initialized via the `init()` in the conditionally included `providers_nofips.go`.
I am quite rusty at Golang - especially internals - but I wonder if there is any chance that `init()` on the `providers_nofips` might be executed after the `init` [here](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go#L107) in a non-deterministic behavior.
Contributor guide
Research direction
Read libbeat/processors/add_cloud_metadata/providers.go, providers_nofips.go, and add_cloud_metadata.go, focusing on provider registration and parallel probing. Reproduce the missing Azure metadata after an Azure VM restart and compare global_processors.txt before and after restarting Elastic Agent; done means Azure metadata is consistently attached without an agent restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100