influxdata / influxdata/telegraf

Issue: Memory grows unbounded when using disk buffer + Stackdriver rejects

Open
#17,946 0 comments 0 reactions 1 assignee View on GitHub

@srebhan is already working on this.

Since Jun 8, 2026.

bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[agent]
interval = "5s"
flush_interval = "5s"
metric_batch_size = 1000 # We also saw this when set to 5000
metric_buffer_limit = 1000000
buffer_strategy = "disk"
buffer_directory = "$TELEGRAF_BUFFER_DIRECTORY"
debug = true

[[inputs.socket_listener]]
service_address = "tcp://:2878"
data_format = "wavefront"

[[inputs.http_listener_v2]]
service_address = ":2877"
data_format = "wavefront"
paths = ["/", "/report","/api"]

[[outputs.stackdriver]]
namepass = ["prod_*","qa_*","dev_*","tas_*"]
project = "MASKED" # Masked for obvious reasons
resource_type = "prometheus_target"
metric_name_format = "official"
metric_data_type = "double"
metric_type_prefix = "prometheus.googleapis.com"
tags_as_resource_label = ["instance"] # inherit instance label as resource label
[outputs.stackdriver.resource_labels]
cluster = "onprem"
instance = "metrics.ioq1.server.com"
job = "${INSTANCE_HOSTNAME}"
location = "us-east1-b"
namespace = "telegraf-receiver"
[outputs.stackdriver.tagpass]
_destination = ["gmp"]
```

### Logs from Telegraf

```text
W! [outputs.stackdriver] Unable to write to Stackdriver - dropping metrics:
rpc error: code = InvalidArgument desc = ... Points must be written in order ...
```

### System info

Telegraf 1.36.3 - Ubuntu 22.04.5 LTS - Docker 28.5.1

### Docker

_No response_

### Steps to reproduce

Using disk buffer strategy
1️⃣ Send ~1000 metrics per second into Telegraf via socket/http listeners
2️⃣ Metrics that violate Google metric requirements trigger Stackdriver InvalidArgument errors, and metrics are dropped from the plugin (expected)
3️⃣ Memory grows linearly over time while buffer balloons, as if it's not sending anything at all, but it clearly is by what shows in the logs
4️⃣ Eventually RSS exceeds container memory limit and Telegraf is OOM-killed

As an aside it looks like metrics are being retried because after I let it run for a while the buffer gets massive. Roughly 22 million points accumulated in the buffer in past ~18 hours. The other agents that are behind the same (using memory strat) LB VIP have buffers at ~20-80K.

### Expected behavior

It should work more or less the same as the default memory strategy, but using disk as the buffer instead.

### Actual behavior

Depending on Telegraf ingress volume and (I think) how often the metrics are rejected by Google causes memory consumption to slowly increase:

13:48 | restart w/ disk buffer enabled | 140 MB
-- | -- | --
14:25 | sustained rejects | 1.75 GB
15:06 | continued workload | 3.93 GB
+18 hours uptime | ongoing | 17.78 GB (56% of 31.34 GB node)

Telegraf Heap telemetry and docker stats show a nearly linear upward trend which appear to be tied to write rejects.

Visual evidence

📈 Metrics Gathered vs Written (rejects increasing)
📈 Stackdriver buffer size increasing exponentially
📈 heap_in_use_bytes rising in parallel

Image

Image

Image

### Additional info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.