elastic / elastic/beats

add_cloud_metadata: flaky behavior for azure provider reproduced on Azure VMs after VM restart

Open
#48,536 2 comments 2 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

As per user report, we have been able to reproduce the fact the `add_cloud_metadata` is not always attaching the Azure metadata when running on a Azure VM, especially after a VM restart

Restarting Elastic Agent can randomly make it re-add the metadata again.

The `add_cloud_metadata` are implemented without any retry, so if the first attempt fails (e.g. while the VM is still lacking connectivity/access to network), the metadata will not be attached.
Still, in our tests we verified that changing the Elastic Agent Systemd unit to REQUIRE network as follows didn't help.
```
[Unit]
Description=Elastic Agent is a unified agent to observe, monitor and protect your system.
ConditionFileIsExecutable=/usr/bin/elastic-agent
After=network-online.target
Wants=network-online.target
```

The fact the processor is not attaching metadata can be seen in the EA diagnostic, under `global_processors.txt`:
```
components/beat-metrics-monitoring/global_processors.txt:add_cloud_metadata={}
components/filestream-monitoring/global_processors.txt:add_cloud_metadata={}
components/http-metrics-monitoring/global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
components/log-default/global_processors.txt:add_cloud_metadata={}
components/system-metrics-default/global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
```

Once EA is restarted, it might get all the components with the:
```
components/...all of them... /global_processors.txt:add_cloud_metadata={"cloud":{"account":{"id":"...
```

This has been observed on Beats & Elastic Agent 8.19.x

### What to expect as a fix

The `add_cloud_metadata` processor should not be flaky after a VM restart. The resulting data produced with this provider could prevent security or observability alerts/detection rules to fail as some crucial metadata is missing.

### Possible reason?

Looking at the code, it seems we register several providers (based on config or otherwise we go to [defaults](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers.go#L58)) and then we do a [parallel probing](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers.go#L180).

I just noticed `azure` provider is only part of the non-fips build [here](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/providers_nofips.go#L27).

In 8.18, `azure` was statically initialized https://github.com/elastic/beats/blob/v8.18.0/libbeat/processors/add_cloud_metadata/providers.go

On 8.19, via https://github.com/elastic/beats/commit/0ea1263cf1d63509e53516daffdc5737b63b7c09 (due to FIPS builds), it became conditionally initialized via the `init()` in the conditionally included `providers_nofips.go`.

I am quite rusty at Golang - especially internals - but I wonder if there is any chance that `init()` on the `providers_nofips` might be executed after the `init` [here](https://github.com/elastic/beats/blob/c7cf7213300557b19c72bddd70318f986d2a0aed/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go#L107) in a non-deterministic behavior.

Contributor guide

Open the contributing guide

Research direction

Read libbeat/processors/add_cloud_metadata/providers.go, providers_nofips.go, and add_cloud_metadata.go, focusing on provider registration and parallel probing. Reproduce the missing Azure metadata after an Azure VM restart and compare global_processors.txt before and after restarting Elastic Agent; done means Azure metadata is consistently attached without an agent restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.