facebook / facebook/rocksdb

Use file descriptors and not c-lib FILE pointers on xBSD

Open
#8,184 5 comments 2 reactions 0 assignees View on GitHub
up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

### Expected behavior
RocksDB should be able to open more then 32767 files on xBSD.

### Actual behavior
RocksDB will fail when opening file pointer with number higher then 32767. This is due to following code in FreeBSD (and other xBSD incarnation) C-Lib: https://github.com/freebsd/freebsd-src/blob/main/lib/libc/stdio/fdopen.c#L64-L67.

And this is the line of RocksDB code that invokes fdopen: https://github.com/facebook/rocksdb/blob/master/env/fs_posix.cc#L185

The problem is not caused by OS limits, this is the issue with xBSD not being able to handle a FP with number higher then 32767.

### Steps to reproduce the behavior
The problem has been reproduced on FreeBSD v12.2 and v13.0 using latest RocksDB master and large dataset with > 32767 files. Any attempt to open such dataset will result in EMFILE (too many open files) error.

### Suggested change
On FreeBSD Forum change from file pointers to file descriptors has been suggested: https://forums.FreeBSD.org/threads/opening-more-then-32767-file-descriptors.79780/post-505682

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.