influxdata / influxdata/telegraf
exec plugin: Environment variables with `${VAR}` syntax fail when appearing after variables with defaults
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
## Summary
When using the `exec` input plugin, environment variables defined in the `environment` array are incorrectly substituted to empty strings when they appear **after** variables with default values in the command string.
## Environment
- **Telegraf Version**: (tested with versions using compose-go v1.20.2)
- **Plugin**: `inputs.exec`
- **OS**: Linux (likely affects all platforms)
## Configuration
```toml
[[inputs.exec]]
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id ${ENTRY_ID}'"
]
environment = [
"SCRIPTS_PATH=/opt/scripts",
"ENTRY_ID=12345"
]
interval = "60m"
timeout = "20m"
data_format = "nagios"
```
## Expected Behavior
The command should be executed as:
```bash
sh -c '/opt/scripts/check.py --file /tmp/file.jpg --sleep 120 --entry_id ${ENTRY_ID}'
```
Then the shell should substitute `${ENTRY_ID}` with `12345` from the environment array.
## Actual Behavior
The command is executed as:
```bash
sh -c '/opt/scripts/check.py --file /tmp/file.jpg --sleep 120 --entry_id '
```
The `${ENTRY_ID}` variable is replaced with an empty string instead of being preserved for shell substitution.
Telegraf logs show:
```
time="2026-03-16T13:04:34Z" level=warning msg="The \"ENTRY_ID\" variable is not set. Defaulting to a blank string."
```
## Root Cause
This is an **upstream bug** in `github.com/compose-spec/compose-go/template` library used by Telegraf for configuration-time environment variable substitution.
**Upstream Issue**: (https://github.com/compose-spec/compose-go/issues/856)
The bug occurs in `template/template.go` at line ~199 where the library fails to preserve custom configuration options during recursive substitution. When a variable with a default (e.g., `${FILE:-/tmp/file.jpg}`) is processed, subsequent variables without defaults are incorrectly handled.
**Affected compose-go versions**: v1.20.2 (Telegraf's current version), v2.10.1 (latest), and likely all versions in between.
## Workarounds
Until the upstream library is fixed and Telegraf updates its dependency, users can work around this issue:
### Option 1: Reorder Variables (Recommended)
Move variables without defaults **before** variables with defaults:
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --entry_id ${ENTRY_ID} --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120}'"
]
```
### Option 2: Remove Braces
Use `$VAR` instead of `${VAR}` for variables without defaults:
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id $ENTRY_ID'"
]
```
### Option 3: Set in System Environment
Set the variable in the system environment before starting Telegraf:
```bash
export ENTRY_ID=12345
telegraf --config telegraf.conf
```
### Option 4: Use Dummy Default
Add a self-referencing default (less elegant):
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id ${ENTRY_ID:-${ENTRY_ID}}'"
]
```
## Steps to Reproduce
1. Create a Telegraf config with the configuration shown above
2. Ensure `ENTRY_ID` is **not** set in the system environment (only in the `environment` array)
3. Start Telegraf
4. Observe that the `--entry_id` parameter is empty in the executed command
## Additional Context
This issue affects any scenario where:
- Variables are defined in the plugin's `environment` array (for runtime substitution)
- Variables are **not** defined in the system environment (at config-load time)
- A variable without a default appears after a variable with a default in the command string
The bug is position-dependent:
- ✅ Works: `${VAR1} ${VAR2:-default}` (variable without default comes first)
- ❌ Fails: `${VAR1:-default} ${VAR2}` (variable without default comes second)
## Proposed Resolution
1. **Short-term**: Document the workarounds in the exec plugin README
2. **Long-term**:
- Monitor the upstream compose-go issue for a fix
- Update Telegraf's dependency to the fixed version once available
- Add regression tests to prevent similar issues
## Related Links
- Upstream bug report: (https://github.com/compose-spec/compose-go/issues/856)
- Telegraf's environment variable substitution: `config/envvar.go`
- Affected plugin: `plugins/inputs/exec/`
## Impact
This affects users who:
- Use the `exec` plugin with environment variables
- Rely on the `environment` array for runtime variable substitution
- Have variables without defaults appearing after variables with defaults
The workarounds are simple but non-obvious, so documentation would help users avoid this issue.
Contributor guide
Research direction
Reproduce the configuration with inputs/exec, then inspect config/envvar.go and the plugins/inputs/exec/ implementation. Compare Telegraf's substitution behavior with compose-go's template/template.go and upstream issue 856. Done means documenting the workaround or updating to a fixed dependency, with regression coverage if the change is made in Telegraf.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- devops, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100