elastic / elastic/beats

[OTel] Unset / blank environment variables are misinterpreted as objects

Open
#45,102 10 comments 0 reactions 0 assignees View on GitHub
bug Stalled Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

When trying to create a configuration that can be overridden with environment variables, I ran into an issue where the `OTel` `envprovider` is breaking the configuration for unset or blank environment variables that are strings.

Example otel config (`es.yml`)

```yml
receivers:
metricbeatreceiver:
metricbeat:
modules:
- module: elasticsearch
hosts: "http://localhost:9200"
# Auth should only specify either API Key or Username/Password
# api_key: ${env:MY_ES_API_KEY:-}
username: ${env:MY_ES_USERNAME:-}
password: ${env:MY_ES_PASSWORD:-}
period: 10s
metricsets:
- node_stats
output:
otelconsumer:
telemetry_types: ["logs"]

exporters:
debug:
verbosity: detailed
sampling_initial: 1

service:
pipelines:
logs:
receivers: [metricbeatreceiver]
exporters: [debug]
```

```
./elastic-agent otel --config es.yml
```

This also fails for the same reason if you set the variables to blank:

```
MY_ES_USERNAME="" MY_ES_PASSWORD="" ./elastic-agent otel --config es.yml
```

Both cases will result in failure that looks like:

> failed to build pipelines: failed to create "metricbeatreceiver" receiver for data type "logs": error creating metricbeatreceiver: error getting metricbeat creator:host parsing failed for elasticsearch-node_stats: 'username' config for module elasticsearch is not a string

This starts successfully if you set them to any value (auth will only succeed if you set it to a valid value of course):

```
MY_ES_USERNAME=x MY_ES_PASSWORD=x ./elastic-agent otel --config es.yml
```

This does not fail because of some weird parsing issue -- instead, the blank variety of values are being provided to the underlying config objects as a struct defined in OTel that looks like a `map` (I forced the Beat to print the value):

```
map[original: value:]
```

While unclear at first, the `original` field's value is the blank string `""`. This "map" clearly comes from:

https://github.com/open-telemetry/opentelemetry-collector/blob/a33fdf3cb7547ce4c825a17d742703e65af7f053/confmap/expand.go#L177-L182

which is the eventual result of:

https://github.com/open-telemetry/opentelemetry-collector/blob/a33fdf3cb7547ce4c825a17d742703e65af7f053/confmap/provider/envprovider/provider.go#L67

This is mostly reproducable with a simple Go test as long as you have `confmap` from OTel (exists in both the Beats and Agent codebases):

```go
func TestRetrieveValue(t *testing.T) {
expected := "" // test fails with ""
// expected := "xyz" // test passes with "xyz"

value, err := confmap.NewRetrievedFromYAML([]byte(expected))

require.NoError(t, err)

str, err := value.AsString()

require.NoError(t, err)
require.Equal(t, str, expected)

conf, err := value.AsRaw()

require.NoError(t, err)
require.Equal(t, conf, expected) // when expected is "", conf is nil
}
```

Contributor guide

Open the contributing guide

Research direction

The issue points to OTel confmap/provider/envprovider/provider.go and confmap/expand.go; start by running the minimal TestRetrieveValue reproduction against the confmap package. Trace how an empty environment value becomes a retrieved object, then verify that unset and blank string variables remain valid strings and the Elasticsearch configuration starts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.