influxdata / influxdata/telegraf

exec plugin: Environment variables with `${VAR}` syntax fail when appearing after variables with defaults

Open
#18,553 1 comment 0 reactions 0 assignees View on GitHub
bug upstream
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

## Summary

When using the `exec` input plugin, environment variables defined in the `environment` array are incorrectly substituted to empty strings when they appear **after** variables with default values in the command string.

## Environment

- **Telegraf Version**: (tested with versions using compose-go v1.20.2)
- **Plugin**: `inputs.exec`
- **OS**: Linux (likely affects all platforms)

## Configuration

```toml
[[inputs.exec]]
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id ${ENTRY_ID}'"
]
environment = [
"SCRIPTS_PATH=/opt/scripts",
"ENTRY_ID=12345"
]
interval = "60m"
timeout = "20m"
data_format = "nagios"
```

## Expected Behavior

The command should be executed as:
```bash
sh -c '/opt/scripts/check.py --file /tmp/file.jpg --sleep 120 --entry_id ${ENTRY_ID}'
```

Then the shell should substitute `${ENTRY_ID}` with `12345` from the environment array.

## Actual Behavior

The command is executed as:
```bash
sh -c '/opt/scripts/check.py --file /tmp/file.jpg --sleep 120 --entry_id '
```

The `${ENTRY_ID}` variable is replaced with an empty string instead of being preserved for shell substitution.

Telegraf logs show:
```
time="2026-03-16T13:04:34Z" level=warning msg="The \"ENTRY_ID\" variable is not set. Defaulting to a blank string."
```

## Root Cause

This is an **upstream bug** in `github.com/compose-spec/compose-go/template` library used by Telegraf for configuration-time environment variable substitution.

**Upstream Issue**: (https://github.com/compose-spec/compose-go/issues/856)

The bug occurs in `template/template.go` at line ~199 where the library fails to preserve custom configuration options during recursive substitution. When a variable with a default (e.g., `${FILE:-/tmp/file.jpg}`) is processed, subsequent variables without defaults are incorrectly handled.

**Affected compose-go versions**: v1.20.2 (Telegraf's current version), v2.10.1 (latest), and likely all versions in between.

## Workarounds

Until the upstream library is fixed and Telegraf updates its dependency, users can work around this issue:

### Option 1: Reorder Variables (Recommended)
Move variables without defaults **before** variables with defaults:
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --entry_id ${ENTRY_ID} --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120}'"
]
```

### Option 2: Remove Braces
Use `$VAR` instead of `${VAR}` for variables without defaults:
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id $ENTRY_ID'"
]
```

### Option 3: Set in System Environment
Set the variable in the system environment before starting Telegraf:
```bash
export ENTRY_ID=12345
telegraf --config telegraf.conf
```

### Option 4: Use Dummy Default
Add a self-referencing default (less elegant):
```toml
commands = [
"sh -c '${SCRIPTS_PATH}/check.py --file ${FILE:-/tmp/file.jpg} --sleep ${SLEEP:-120} --entry_id ${ENTRY_ID:-${ENTRY_ID}}'"
]
```

## Steps to Reproduce

1. Create a Telegraf config with the configuration shown above
2. Ensure `ENTRY_ID` is **not** set in the system environment (only in the `environment` array)
3. Start Telegraf
4. Observe that the `--entry_id` parameter is empty in the executed command

## Additional Context

This issue affects any scenario where:
- Variables are defined in the plugin's `environment` array (for runtime substitution)
- Variables are **not** defined in the system environment (at config-load time)
- A variable without a default appears after a variable with a default in the command string

The bug is position-dependent:
- ✅ Works: `${VAR1} ${VAR2:-default}` (variable without default comes first)
- ❌ Fails: `${VAR1:-default} ${VAR2}` (variable without default comes second)

## Proposed Resolution

1. **Short-term**: Document the workarounds in the exec plugin README
2. **Long-term**:
- Monitor the upstream compose-go issue for a fix
- Update Telegraf's dependency to the fixed version once available
- Add regression tests to prevent similar issues

## Related Links

- Upstream bug report: (https://github.com/compose-spec/compose-go/issues/856)
- Telegraf's environment variable substitution: `config/envvar.go`
- Affected plugin: `plugins/inputs/exec/`

## Impact

This affects users who:
- Use the `exec` plugin with environment variables
- Rely on the `environment` array for runtime variable substitution
- Have variables without defaults appearing after variables with defaults

The workarounds are simple but non-obvious, so documentation would help users avoid this issue.

Contributor guide

Open the contributing guide

Research direction

Reproduce the configuration with inputs/exec, then inspect config/envvar.go and the plugins/inputs/exec/ implementation. Compare Telegraf's substitution behavior with compose-go's template/template.go and upstream issue 856. Done means documenting the workaround or updating to a fixed dependency, with regression coverage if the change is made in Telegraf.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, shell
Domain
devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.