influxdata / influxdata/telegraf
update state more frequently in case telegraf is suddenly shutdown
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Currently state is only stored once at the end of a telegraf run. In cases where telegraf fails catastrophically, this means loss of state.
Running a telegraf container with docker-compose
docker-compose.yaml
```yaml
version: '3'
services:
telegraf:
image: telegraf:latest
volumes:
- ./files/etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- ./file.log:/nas/log/file.log:ro
- ./telegraf_state.txt:/nas/log/cursors/telegraf_state.txt:wro
command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf"]
```
telegraf.conf
```toml
[global_tags]
platform = "evlv"
[agent]
interval = "30s"
flush_interval = "60s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_jitter = "5s"
precision = ""
debug = true
hostname = ""
omit_hostname = false
statefile = "/nas/log/cursors/telegraf_state.txt"
[[inputs.tail]]
files = ["/nas/log/file.log"]
from_beginning = false
data_format = "grok"
grok_patterns = ["%{GREEDYDATA:log}"]
[[outputs.file]]
files = ["stdout"]
```
telegraf.txt
```json
{}
```
The file.log is empty to start.
Suddenly killing the container causes no state to be recorded
Adding log lines before starting it back up causes those log lines to be lost
### Expected behavior
When the telegraf container has read from a file via tail plugin and is suddenly deleted without a safe shutdown, state is recorded.
### Actual behavior
When the telegraf container has read from a file via tail plugin and is suddenly deleted without a safe shutdown, state is not recorded.
### Additional info
_No response_
Contributor guide
Research direction
Start by reproducing the abrupt container deletion with the provided docker-compose.yaml and telegraf.conf, then trace how the tail input records the configured statefile. Check how state is persisted during normal reading rather than only at shutdown. Done means a sudden deletion preserves enough state that subsequently appended log lines are not lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100