influxdata / influxdata/telegraf
Disk buffer inode leak (orphaned files) after configuration change
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
# $ mkdir -p /tmp/tg/buffer
# $ watch 'ls -lR /tmp/tg/buffer' # in one terminal
# $ cp /path/to/this/config /tmp/tg/telegraf.conf # in another terminal
# $ telegraf --config /tmp/tg/telegraf.conf
# ...see subdir in /tmp/tg/buffer with a file that grows and shrinks (good)...
# ...ctrl-c to stop telegraf, it flushes the buffer, 0s it and stops (good)...
# $ telegraf --config /tmp/tg/telegraf.conf
# ...restarting with same config uses same parent dir for buffer file...
# ...ctrl-c to stop telegraf, it flushes the buffer, 0s it and stops (good)...
#
# Now adjust output plugin to use a different valud for 'rotation_max_size'
# $ telegraf --config /tmp/tg/telegraf.conf
# ...restarting creates different buffer subdir, leaving the other behind...
# Configuration for telegraf agent
[agent]
# defaults from telegraf.conf.sample
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_jitter = "0s"
precision = "0s"
# reproducer setup
interval = "1s"
flush_interval = "1s"
buffer_strategy = "disk"
buffer_directory = "/tmp/tg/buffer"
# Setup an output
[[outputs.file]]
files = ["/tmp/tg/out1.txt"]
data_format = "influx"
# change this and restart telegraf to have it use a new buffer subdir
rotation_max_size = "10MB"
# Get some metrics from the system
[[inputs.cpu]]
# defaults from telegraf.conf.sample
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
core_tags = false
```
### Logs from Telegraf
```text
# first start
$ telegraf --config ./telegraf.conf --debug
2026-01-30T12:03:04Z W! Strict environment variable handling will be the new default starting with v1.38.0! If your configuration works with strict handling or you don't use environment variables it is safe to ignore this warning. Otherwise please explicitly add the --non-strict-env-handling flag!
2026-01-30T12:03:04Z I! Loading config: ./telegraf.conf
2026-01-30T12:03:04Z W! Using disk-write-through buffer strategy for plugin outputs.file, this is an experimental feature
2026-01-30T12:03:04Z I! Starting Telegraf 1.37.1 brought to you by InfluxData the makers of InfluxDB
2026-01-30T12:03:04Z I! Available plugins: 243 inputs, 9 aggregators, 35 processors, 26 parsers, 67 outputs, 8 secret-stores
2026-01-30T12:03:04Z I! Loaded inputs: cpu
2026-01-30T12:03:04Z I! Loaded aggregators:
2026-01-30T12:03:04Z I! Loaded processors:
2026-01-30T12:03:04Z I! Loaded secretstores:
2026-01-30T12:03:04Z I! Loaded outputs: file
2026-01-30T12:03:04Z I! Tags enabled: host=tg-repro
2026-01-30T12:03:04Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:"tg-repro", Flush Interval:1s
2026-01-30T12:03:04Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2026-01-30T12:03:04Z D! [agent] Initializing plugins
2026-01-30T12:03:04Z D! [agent] Connecting outputs
2026-01-30T12:03:04Z D! [agent] Attempting connection to [outputs.file]
2026-01-30T12:03:04Z D! [agent] Successfully connected to outputs.file
2026-01-30T12:03:04Z D! [agent] Starting service inputs
2026-01-30T12:03:05Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:03:06Z D! [outputs.file] Wrote batch of 21 metrics in 163.305µs
2026-01-30T12:03:06Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:03:07Z D! [outputs.file] Wrote batch of 21 metrics in 266.342µs
2026-01-30T12:03:07Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:03:08Z D! [outputs.file] Wrote batch of 21 metrics in 266.88µs
2026-01-30T12:03:08Z D! [outputs.file] Buffer fullness: 0 metrics
^C2026-01-30T12:03:09Z D! [agent] Stopping service inputs
2026-01-30T12:03:09Z D! [agent] Input channel closed
2026-01-30T12:03:09Z I! [agent] Hang on, flushing any cached metrics before shutdown
2026-01-30T12:03:09Z D! [outputs.file] Wrote batch of 21 metrics in 179.104µs
2026-01-30T12:03:09Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:03:09Z I! [agent] Stopping running outputs
2026-01-30T12:03:09Z D! [agent] Stopped Successfully
# restart with no config changes
$ telegraf --config ./telegraf.conf --debug
2026-01-30T12:04:55Z W! Strict environment variable handling will be the new default starting with v1.38.0! If your configuration works with strict handling or you don't use environment variables it is safe to ignore this warning. Otherwise please explicitly add the --non-strict-env-handling flag!
2026-01-30T12:04:55Z I! Loading config: ./telegraf.conf
2026-01-30T12:04:55Z W! Using disk-write-through buffer strategy for plugin outputs.file, this is an experimental feature
2026-01-30T12:04:55Z I! Starting Telegraf 1.37.1 brought to you by InfluxData the makers of InfluxDB
2026-01-30T12:04:55Z I! Available plugins: 243 inputs, 9 aggregators, 35 processors, 26 parsers, 67 outputs, 8 secret-stores
2026-01-30T12:04:55Z I! Loaded inputs: cpu
2026-01-30T12:04:55Z I! Loaded aggregators:
2026-01-30T12:04:55Z I! Loaded processors:
2026-01-30T12:04:55Z I! Loaded secretstores:
2026-01-30T12:04:55Z I! Loaded outputs: file
2026-01-30T12:04:55Z I! Tags enabled: host=tg-repro
2026-01-30T12:04:55Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:"tg-repro", Flush Interval:1s
2026-01-30T12:04:55Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2026-01-30T12:04:55Z D! [agent] Initializing plugins
2026-01-30T12:04:55Z D! [agent] Connecting outputs
2026-01-30T12:04:55Z D! [agent] Attempting connection to [outputs.file]
2026-01-30T12:04:55Z D! [agent] Successfully connected to outputs.file
2026-01-30T12:04:55Z D! [agent] Starting service inputs
2026-01-30T12:04:56Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:04:57Z D! [outputs.file] Wrote batch of 21 metrics in 239.987µs
2026-01-30T12:04:57Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:04:58Z D! [outputs.file] Wrote batch of 21 metrics in 241.047µs
2026-01-30T12:04:58Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:04:59Z D! [outputs.file] Wrote batch of 21 metrics in 262.429µs
2026-01-30T12:04:59Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:05:00Z D! [outputs.file] Wrote batch of 21 metrics in 204.819µs
2026-01-30T12:05:00Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:05:01Z D! [outputs.file] Wrote batch of 21 metrics in 220.018µs
2026-01-30T12:05:01Z D! [outputs.file] Buffer fullness: 0 metrics
^C2026-01-30T12:05:01Z D! [agent] Stopping service inputs
2026-01-30T12:05:01Z D! [agent] Input channel closed
2026-01-30T12:05:01Z I! [agent] Hang on, flushing any cached metrics before shutdown
2026-01-30T12:05:01Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:05:01Z I! [agent] Stopping running outputs
2026-01-30T12:05:01Z D! [agent] Stopped Successfully
# restart with 'rotation_max_size' set differently
$ telegraf --config ./telegraf.conf --debug
2026-01-30T12:06:36Z W! Strict environment variable handling will be the new default starting with v1.38.0! If your configuration works with strict handling or you don't use environment variables it is safe to ignore this warning. Otherwise please explicitly add the --non-strict-env-handling flag!
2026-01-30T12:06:36Z I! Loading config: ./telegraf.conf
2026-01-30T12:06:36Z W! Using disk-write-through buffer strategy for plugin outputs.file, this is an experimental feature
2026-01-30T12:06:36Z I! Starting Telegraf 1.37.1 brought to you by InfluxData the makers of InfluxDB
2026-01-30T12:06:36Z I! Available plugins: 243 inputs, 9 aggregators, 35 processors, 26 parsers, 67 outputs, 8 secret-stores
2026-01-30T12:06:36Z I! Loaded inputs: cpu
2026-01-30T12:06:36Z I! Loaded aggregators:
2026-01-30T12:06:36Z I! Loaded processors:
2026-01-30T12:06:36Z I! Loaded secretstores:
2026-01-30T12:06:36Z I! Loaded outputs: file
2026-01-30T12:06:36Z I! Tags enabled: host=tg-repro
2026-01-30T12:06:36Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:"tg-repro", Flush Interval:1s
2026-01-30T12:06:36Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2026-01-30T12:06:36Z D! [agent] Initializing plugins
2026-01-30T12:06:36Z D! [agent] Connecting outputs
2026-01-30T12:06:36Z D! [agent] Attempting connection to [outputs.file]
2026-01-30T12:06:36Z D! [agent] Successfully connected to outputs.file
2026-01-30T12:06:36Z D! [agent] Starting service inputs
2026-01-30T12:06:37Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:06:38Z D! [outputs.file] Wrote batch of 21 metrics in 226.612µs
2026-01-30T12:06:38Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:06:39Z D! [outputs.file] Wrote batch of 21 metrics in 161.436µs
2026-01-30T12:06:39Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:06:40Z D! [outputs.file] Wrote batch of 21 metrics in 186µs
2026-01-30T12:06:40Z D! [outputs.file] Buffer fullness: 0 metrics
^C2026-01-30T12:06:40Z D! [agent] Stopping service inputs
2026-01-30T12:06:40Z D! [agent] Input channel closed
2026-01-30T12:06:40Z I! [agent] Hang on, flushing any cached metrics before shutdown
2026-01-30T12:06:40Z D! [outputs.file] Buffer fullness: 0 metrics
2026-01-30T12:06:40Z I! [agent] Stopping running outputs
2026-01-30T12:06:40Z D! [agent] Stopped Successfully
```
### System info
Telegraf 1.37.1, Ubuntu 24.04 amd64
### Docker
_No response_
### Steps to reproduce
1. Create a work dir: `mkdir -p /tmp/tg/buffer`
2. Create the telegraf configuration in `/tmp/tg/telegraf.conf` (see above)
3. In one terminal: `watch 'ls -lR /tmp/tg/buffer'`
4. In another terminal, start telegraf: `telegraf --config /tmp/tg/telegraf.conf --debug`
* see subdir in /tmp/tg/buffer with a file that grows and shrinks (good)
```
$ watch 'ls -lR /tmp/tg/buffer
Every 2.0s: ls -lR /tmp/tg/buffer/ tg-repro: Fri Jan 30 06:04:28 2026
/tmp/tg/buffer/:
total 4
drwxr-x--- 2 jamie jamie 4096 Jan 30 06:03 d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed
/tmp/tg/buffer/d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed:
total 0
-rw-r----- 1 jamie jamie 0 Jan 30 06:03 00000000000000000085
```
* ctrl-c to stop telegraf, it flushes the buffer, zeroes it and stops (good)
* run `telegraf --config /tmp/tg/telegraf.conf --debug` again; uses same parent dir for buffer file (fine)
```
$ watch 'ls -lR /tmp/tg/buffer
Every 2.0s: ls -lR /tmp/tg/buffer/ tg-repro: Fri Jan 30 06:05:52 2026
/tmp/tg/buffer/:
total 4
drwxr-x--- 2 jamie jamie 4096 Jan 30 06:05 d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed
/tmp/tg/buffer/d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed:
total 0
-rw-r----- 1 jamie jamie 0 Jan 30 06:05 00000000000000000190
```
* ctrl-c to stop telegraf, it flushes the buffer, zeroes it and stops (good)
5. Adjust output plugin to use a different value for 'rotation_max_size' (eg, '1M')
6. Start telegraf: `telegraf --config /tmp/tg/telegraf.conf --debug`
* see different subdir in /tmp/tg/buffer, but original subdir and (zeroed) buffer file still exist
```
$ watch 'ls -lR /tmp/tg/buffer
Every 2.0s: ls -lR /tmp/tg/buffer/ tg-repro: Fri Jan 30 06:07:02 2026
/tmp/tg/buffer/:
total 8
drwxr-x--- 2 jamie jamie 4096 Jan 30 06:06 0b0bad712f7010d9ca36939c54e5f2d5a35758740d2fd479247664781c14ac70
drwxr-x--- 2 jamie jamie 4096 Jan 30 06:05 d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed
/tmp/tg/buffer/0b0bad712f7010d9ca36939c54e5f2d5a35758740d2fd479247664781c14ac70:
total 0
-rw-r----- 1 jamie jamie 0 Jan 30 06:06 00000000000000000064
/tmp/tg/buffer/d94db58cde7edecc31406bbe392809fc15562735a366adc23b110dc317d5ebed:
total 0
-rw-r----- 1 jamie jamie 0 Jan 30 06:05 00000000000000000190
```
### Expected behavior
`telegraf` should not orphan buffer files and their parent directories after configuration changes.
`telegraf` knows to zero out the file on clean shutdown, perhaps it can remove the buffer file and its parent directory (if it is empty) on shutdown? Alternatively, `telegraf` could on start look for directories in the `buffer_directory` that only have zeroed out files in them, then remove the files and the directories (if empty after zeroed file removals).
### Actual behavior
`telegraf` orphans buffer files and their parent directories after configuration changes (this is a file inode leak).
### Additional info
_No response_
Contributor guide
Research direction
Reproduce the behavior with /tmp/tg/telegraf.conf, the disk buffer settings, and outputs.file, changing rotation_max_size between runs. Locate the disk-buffer and outputs.file entry points, then trace how the buffer directory is selected and cleaned up during startup and shutdown. Done means changing the configuration does not leave the previous empty buffer directory behind while the active buffer continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100