influxdata / influxdata/telegraf

Remotefile output plugin destroys robustness against intermittent internet issues

Open
#18,046 3 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
# This file uses the mock plugin to create dummy metrics. It then uploads
# the metrics to an S3 bucket using the remotefile output plugin.

# Mock input plugin - generates synthetic test metrics
[[inputs.mock]]
metric_name = "demo_metrics"

[[inputs.mock.random]]
name = "humidity"
min = 40.0
max = 60.0

# Remotefile output plugin - sends metrics to S3
[[outputs.remotefile]]
remote = 's3,provider=AWS,access_key_id=YOUR_ACCESS_KEY,secret_access_key=YOUR_SECRET_KEY,region=us-east-1:your-bucket-name'
files = ['{{.Name}}-{{.Time.Format "2006-01-02"}}.log']
cache_max_size = "100MB"

# Forget files after not being touched for longer than the given time
# Useful to prevent memory leaks when using time-based filenames
# Example: "24h" to forget files after 24 hours
forget_files_after = "24h"

## Data format to output
data_format = "json"

# Create another output that writes the metrics to stdout
[[outputs.file]]
files = ["stdout"]
data_format = "json"
```

### Logs from Telegraf

```text
2025-11-22T11:13:51Z I! Loading config: s3-demo.conf
2025-11-22T11:13:51Z I! Starting Telegraf 1.36.4 brought to you by InfluxData the makers of InfluxDB
2025-11-22T11:13:51Z I! Available plugins: 238 inputs, 9 aggregators, 35 processors, 26 parsers, 65 outputs, 5 secret-stores
2025-11-22T11:13:51Z I! Loaded inputs: mock
2025-11-22T11:13:51Z I! Loaded aggregators:
2025-11-22T11:13:51Z I! Loaded processors:
2025-11-22T11:13:51Z I! Loaded secretstores:
2025-11-22T11:13:51Z I! Loaded outputs: file remotefile
2025-11-22T11:13:51Z I! Tags enabled: host=Daniels-MacBook-Pro-2.local
2025-11-22T11:13:51Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"Daniels-MacBook-Pro-2.local", Flush Interval:10s
2025-11-22T11:13:51Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-11-22T11:13:51Z D! [agent] Initializing plugins
2025-11-22T11:13:51Z D! [agent] Connecting outputs
2025-11-22T11:13:51Z D! [agent] Attempting connection to [outputs.remotefile]
2025-11-22T11:13:51Z E! [agent] Failed to connect to [outputs.remotefile], retrying in 15s, error was "operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"https://your-bucket-name.s3.us-east-1.amazonaws.com/?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=\": dial tcp: lookup your-bucket-name.s3.us-east-1.amazonaws.com: no such host"
2025-11-22T11:14:06Z E! [telegraf] Error running agent: connecting output outputs.remotefile: error connecting to output "outputs.remotefile": operation error S3: ListObjectsV2, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://your-bucket-name.s3.us-east-1.amazonaws.com/?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=": dial tcp: lookup your-bucket-name.s3.us-east-1.amazonaws.com: no such host
```

### System info

Telegraf 1.36.4 (MacOS in this case, but also tested on Linux)

### Docker

_No response_

### Steps to reproduce

1. Use the `remotefile` output with AWS S3 in a workflow
2. Turn off internet access
3. Start the workflow
4. Telegraf process exits with error, preventing any other workflow from running or data from being buffered

### Expected behavior

If a telegraf output plugin is not currently usable (e.g. missing working internet), the data should be buffered so that it can be uploaded when the connection is back. This is how all other telegraf output plugins work.

I would expect the plugin to just buffer up the files locally and then send them once internet is back. This is how it works if I start telegraf while I have internet, and then loose internet.

### Actual behavior

With the `remotefile` plugin, this does not seem to be the case. In some usages, we have seen that the telegraf process freezes while starting if the `remotefile` plugin can't access AWS S3, in this simple isolated experiment that I uploaded in this issue ticket, telegraf is instead exiting with an error.

This is a common case, since a power outage might reboot the computer, and when it starts again, I might not have internet. This time is critical to capture, it is simply not acceptable that telegraf fails to start and is in a crash loop without collecting metrics until I get internet again.

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the remotefile output plugin's startup connection to AWS S3, using the provided telegraf.conf and offline reproduction steps. Done means Telegraf remains running without internet, buffers metrics locally, and uploads them after connectivity returns instead of exiting or freezing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.