influxdata / influxdata/telegraf

update state more frequently in case telegraf is suddenly shutdown

Open
#14,566 1 comment 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Currently state is only stored once at the end of a telegraf run. In cases where telegraf fails catastrophically, this means loss of state.

Running a telegraf container with docker-compose

docker-compose.yaml
```yaml
version: '3'
services:
telegraf:
image: telegraf:latest
volumes:
- ./files/etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
- ./file.log:/nas/log/file.log:ro
- ./telegraf_state.txt:/nas/log/cursors/telegraf_state.txt:wro
command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf"]
```

telegraf.conf
```toml
[global_tags]
platform = "evlv"

[agent]
interval = "30s"
flush_interval = "60s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_jitter = "5s"
precision = ""

debug = true

hostname = ""
omit_hostname = false
statefile = "/nas/log/cursors/telegraf_state.txt"

[[inputs.tail]]
files = ["/nas/log/file.log"]
from_beginning = false
data_format = "grok"
grok_patterns = ["%{GREEDYDATA:log}"]

[[outputs.file]]
files = ["stdout"]
```

telegraf.txt
```json
{}
```

The file.log is empty to start.

Suddenly killing the container causes no state to be recorded
Screenshot 2024-01-11 at 12 16 44 PM

Screenshot 2024-01-11 at 12 15 59 PM

Adding log lines before starting it back up causes those log lines to be lost
Screenshot 2024-01-11 at 12 23 34 PM
Screenshot 2024-01-11 at 12 25 30 PM

### Expected behavior

When the telegraf container has read from a file via tail plugin and is suddenly deleted without a safe shutdown, state is recorded.

### Actual behavior

When the telegraf container has read from a file via tail plugin and is suddenly deleted without a safe shutdown, state is not recorded.

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the abrupt container deletion with the provided docker-compose.yaml and telegraf.conf, then trace how the tail input records the configured statefile. Check how state is persisted during normal reading rather than only at shutdown. Done means a sudden deletion preserves enough state that subsequently appended log lines are not lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.