influxdata / influxdata/influxdb
Low ulimits cause bad experiences for some users on OSS 2.x
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Proposal:__
We should mitigate problems caused by low ulimits. Ideas:
* Reduce the number of shards by default (so fewer files are created by default)
* Add ulimits to `/metrics` endpoint
* Warn in the logs (and UI?) when file ulimits are low
__Current behavior:__
Users start up a bucket with a 30 day retention policy and get daily shards. Many linux distros set the max `ulimit -n` to 1024 . 30 days * ~20 files per shard when TSI is enabled creates ~600 files, so with their first bucket users are right up against their ulimits.
__Desired behavior:__
TBD
__Alternatives considered:__
Document that users should increase their ulimits as part of setup
__Use case:__
Many users get inscrutable "too many files" errors in the logs, which causes much frustration and also data issues as new files cannot be written.
Contributor guide
Research direction
The issue names no source file, test, or entry point. Start by reproducing a 30-day bucket with low file limits and inspect shard file creation, `/metrics`, and log behavior; a contribution is complete only after maintainers define and validate a concrete mitigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- databases, observability, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100