influxdata / influxdata/influxdb
[2.7.11] OOM Error after "Error while freeing cold shard resources"
@davidby-influx is already working on this.
Since Nov 24, 2025.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
We are running InfluxDB 2.7.11 on our Azure Kubernetes Cluster. Yesterday I observed a strange behaviour which led to an OOM error and pod restart. Down below I attached the logs and some pictures of our utilization logging in grafana and the azure portal.
__Expected behaviour:__
No OOM error.
__Actual behaviour:__
From what I can tell from the logs it started with the following errors:
`ts=2025-11-11T10:26:03.289556Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4518`
`ts=2025-11-11T10:26:03.289998Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4488`
`ts=2025-11-11T10:26:03.290257Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4258`
`ts=2025-11-11T10:26:03.290273Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4757`
`ts=2025-11-11T10:26:03.290280Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4698`
`ts=2025-11-11T10:26:03.290450Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4548`
`ts=2025-11-11T10:26:03.290471Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4458`
`ts=2025-11-11T10:26:03.290598Z lvl=warn msg="Error while freeing cold shard resources" log_id=0xDCNLOW000 service=storage-engine service=store error="engine is closed" db_shard_id=4428`
Which then led to more than a couple of errors like:
`ts=2025-11-11T10:27:00.980353Z lvl=info msg="Error exhausting result iterator" log_id=0xDCNLOW000 service=task-executor error="engine is closed" name=to3`
This went on for hours without us noticing (10am till 10pm).
At the end write errors to the DB occured:
`ts=2025-11-11T22:06:08.108573Z lvl=error msg="Unable to write gathered points" log_id=0xDCNLOW000 service=scraper scraper-name=InfluxdbLogs error=timeout`
This went on for another hour before finally our telegraf client started to get timeout errors as well and nothing could be written to the DB (11:06 pm). At this point the influxdb timed out every external connection, but the internal log scraper was still running. Before this point telegraf was still able to connect and write points.
While the scraper was unable to write, the RAM usage went up (buffering all the failed writes?) until finally hitting the configured limit and the pod got terminated. This took another ~ 16 hours.
As you can see from the grafana screenshot there was no sudden peak in usage and after the restart everything went on running fine without any interaction on our side. I still want to make sure I am not missing anything here and would like to further debug the issue. Any recommendations?
__Environment info:__
Influxdb 2.7.11 on AKS with kubernetes version 1.31.3
Persistant Volumens are azure-standard-ssd type
DB size 20 GB (128GB total space)
Bucket cardinality (Logs/Data) both under 10.000
AKS logs showing growing ram usage:
Grafana screenshot showing gap in influxdb logging and system usage before/after:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.