influxdata / influxdata/influxdata-docker
Too many open files since containerd 2.0
- Dominant language
- Shell
- Stars
- 364
- Forks
- 255
- Avg merge
- 3h 18m
- Merged PRs (30d)
- 7
Description
Containerd 2.0 removed `LimitNOFILE=infinity` from its service file, resulting in allowing only 1024 open files in the docker container. See https://github.com/containerd/containerd/pull/8924
This leeds to `msg="Error writing snapshot from compactor" engine=tsm1 op_name=tsm1_cache_snapshot error="compaction in progress: open /var/lib/influxdb/data/db/retention/333/002671920-000000001.tsm.tmp: too many open files"`. This then triggers https://github.com/influxdata/influxdb/issues/26073, when using a not-uptodate version, resulting in filling up the disk.
The non-docker versions set `LimitNOFILE=65536` in its service file (respectively `ulimit -n 65536` in init.sh). I think this is also needed in `Dockerfile`.
The issue has been seen with influxdb:1.8.10-alpine, but probably applies to all versions.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the Dockerfile with the non-Docker service configuration and init.sh, which set the intended open-file limit to 65536. Verify the resulting limit in an affected container and confirm that compaction no longer hits the 1024-file ceiling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100