influxdata / influxdata/telegraf
Issue: Memory grows unbounded when using disk buffer + Stackdriver rejects
@srebhan is already working on this.
Since Jun 8, 2026.
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[agent]
interval = "5s"
flush_interval = "5s"
metric_batch_size = 1000 # We also saw this when set to 5000
metric_buffer_limit = 1000000
buffer_strategy = "disk"
buffer_directory = "$TELEGRAF_BUFFER_DIRECTORY"
debug = true
[[inputs.socket_listener]]
service_address = "tcp://:2878"
data_format = "wavefront"
[[inputs.http_listener_v2]]
service_address = ":2877"
data_format = "wavefront"
paths = ["/", "/report","/api"]
[[outputs.stackdriver]]
namepass = ["prod_*","qa_*","dev_*","tas_*"]
project = "MASKED" # Masked for obvious reasons
resource_type = "prometheus_target"
metric_name_format = "official"
metric_data_type = "double"
metric_type_prefix = "prometheus.googleapis.com"
tags_as_resource_label = ["instance"] # inherit instance label as resource label
[outputs.stackdriver.resource_labels]
cluster = "onprem"
instance = "metrics.ioq1.server.com"
job = "${INSTANCE_HOSTNAME}"
location = "us-east1-b"
namespace = "telegraf-receiver"
[outputs.stackdriver.tagpass]
_destination = ["gmp"]
```
### Logs from Telegraf
```text
W! [outputs.stackdriver] Unable to write to Stackdriver - dropping metrics:
rpc error: code = InvalidArgument desc = ... Points must be written in order ...
```
### System info
Telegraf 1.36.3 - Ubuntu 22.04.5 LTS - Docker 28.5.1
### Docker
_No response_
### Steps to reproduce
Using disk buffer strategy
1️⃣ Send ~1000 metrics per second into Telegraf via socket/http listeners
2️⃣ Metrics that violate Google metric requirements trigger Stackdriver InvalidArgument errors, and metrics are dropped from the plugin (expected)
3️⃣ Memory grows linearly over time while buffer balloons, as if it's not sending anything at all, but it clearly is by what shows in the logs
4️⃣ Eventually RSS exceeds container memory limit and Telegraf is OOM-killed
As an aside it looks like metrics are being retried because after I let it run for a while the buffer gets massive. Roughly 22 million points accumulated in the buffer in past ~18 hours. The other agents that are behind the same (using memory strat) LB VIP have buffers at ~20-80K.
### Expected behavior
It should work more or less the same as the default memory strategy, but using disk as the buffer instead.
### Actual behavior
Depending on Telegraf ingress volume and (I think) how often the metrics are rejected by Google causes memory consumption to slowly increase:
13:48 | restart w/ disk buffer enabled | 140 MB
-- | -- | --
14:25 | sustained rejects | 1.75 GB
15:06 | continued workload | 3.93 GB
+18 hours uptime | ongoing | 17.78 GB (56% of 31.34 GB node)
Telegraf Heap telemetry and docker stats show a nearly linear upward trend which appear to be tied to write rejects.
Visual evidence
📈 Metrics Gathered vs Written (rejects increasing)
📈 Stackdriver buffer size increasing exponentially
📈 heap_in_use_bytes rising in parallel
### Additional info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.