influxdata / influxdata/influxdata-docker

Too many open files since containerd 2.0

Open
#847 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
364
Forks
255
Avg merge
3h 18m
Merged PRs (30d)
7

Description

Containerd 2.0 removed `LimitNOFILE=infinity` from its service file, resulting in allowing only 1024 open files in the docker container. See https://github.com/containerd/containerd/pull/8924

This leeds to `msg="Error writing snapshot from compactor" engine=tsm1 op_name=tsm1_cache_snapshot error="compaction in progress: open /var/lib/influxdb/data/db/retention/333/002671920-000000001.tsm.tmp: too many open files"`. This then triggers https://github.com/influxdata/influxdb/issues/26073, when using a not-uptodate version, resulting in filling up the disk.

The non-docker versions set `LimitNOFILE=65536` in its service file (respectively `ulimit -n 65536` in init.sh). I think this is also needed in `Dockerfile`.

The issue has been seen with influxdb:1.8.10-alpine, but probably applies to all versions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the Dockerfile with the non-Docker service configuration and init.sh, which set the intended open-file limit to 65536. Verify the resulting limit in an affected container and confirm that compaction no longer hits the 1024-file ceiling.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.