[OTel] Unset / blank environment variables are misinterpreted as objects
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
When trying to create a configuration that can be overridden with environment variables, I ran into an issue where the `OTel` `envprovider` is breaking the configuration for unset or blank environment variables that are strings.
Example otel config (`es.yml`)
```yml
receivers:
metricbeatreceiver:
metricbeat:
modules:
- module: elasticsearch
hosts: "http://localhost:9200"
# Auth should only specify either API Key or Username/Password
# api_key: ${env:MY_ES_API_KEY:-}
username: ${env:MY_ES_USERNAME:-}
password: ${env:MY_ES_PASSWORD:-}
period: 10s
metricsets:
- node_stats
output:
otelconsumer:
telemetry_types: ["logs"]
exporters:
debug:
verbosity: detailed
sampling_initial: 1
service:
pipelines:
logs:
receivers: [metricbeatreceiver]
exporters: [debug]
```
```
./elastic-agent otel --config es.yml
```
This also fails for the same reason if you set the variables to blank:
```
MY_ES_USERNAME="" MY_ES_PASSWORD="" ./elastic-agent otel --config es.yml
```
Both cases will result in failure that looks like:
> failed to build pipelines: failed to create "metricbeatreceiver" receiver for data type "logs": error creating metricbeatreceiver: error getting metricbeat creator:host parsing failed for elasticsearch-node_stats: 'username' config for module elasticsearch is not a string
This starts successfully if you set them to any value (auth will only succeed if you set it to a valid value of course):
```
MY_ES_USERNAME=x MY_ES_PASSWORD=x ./elastic-agent otel --config es.yml
```
This does not fail because of some weird parsing issue -- instead, the blank variety of values are being provided to the underlying config objects as a struct defined in OTel that looks like a `map` (I forced the Beat to print the value):
```
map[original: value:]
```
While unclear at first, the `original` field's value is the blank string `""`. This "map" clearly comes from:
https://github.com/open-telemetry/opentelemetry-collector/blob/a33fdf3cb7547ce4c825a17d742703e65af7f053/confmap/expand.go#L177-L182
which is the eventual result of:
https://github.com/open-telemetry/opentelemetry-collector/blob/a33fdf3cb7547ce4c825a17d742703e65af7f053/confmap/provider/envprovider/provider.go#L67
This is mostly reproducable with a simple Go test as long as you have `confmap` from OTel (exists in both the Beats and Agent codebases):
```go
func TestRetrieveValue(t *testing.T) {
expected := "" // test fails with ""
// expected := "xyz" // test passes with "xyz"
value, err := confmap.NewRetrievedFromYAML([]byte(expected))
require.NoError(t, err)
str, err := value.AsString()
require.NoError(t, err)
require.Equal(t, str, expected)
conf, err := value.AsRaw()
require.NoError(t, err)
require.Equal(t, conf, expected) // when expected is "", conf is nil
}
```
Contributor guide
Research direction
The issue points to OTel confmap/provider/envprovider/provider.go and confmap/expand.go; start by running the minimal TestRetrieveValue reproduction against the confmap package. Trace how an empty environment value becomes a retrieved object, then verify that unset and blank string variables remain valid strings and the Elasticsearch configuration starts successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100