Dotnet on OSX is limited to 10240 file handles

Open
#112,726 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
50/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
cpp, csharp, macos
Domain
operating-systems

Research direction

Start with the three limit-setting lines in src/coreclr/pal/src/init/pal.cpp and run the C# reproduction that opens 30,000 files on macOS. Trace how the macOS OPEN_MAX and setrlimit behavior affect the runtime, then verify that the released build can exceed 10,240 open files without the reported failure.

Written by the indexing model from the issue text.

Description

area-VM-coreclr os-mac-os-x
Description

Dotnet on mac is limited to only 10240 file handles, this occurs because:

https://github.com/dotnet/runtime/blob/eab352d9b4339a912456f3577b7dd6f21c0c3ef6/src/coreclr/pal/src/init/pal.cpp#L964-L971

on OS X15.2 OPEN_MAX is defined as:

100:#define OPEN_MAX                10240   /* max open files per process - todo, make a config option? */

I got that from /Library/Developer/CommandLineTools/SDKs/MacOSX15.2.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/syslimits.h, another mac user got the same line from sys/resource.h.

The man page for setrlimit mentions:

COMPATIBILITY
setrlimit() now returns with errno set to EINVAL in places that historically succeeded. It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE. Use "rlim_cur = min(OPEN_MAX, rlim_max)".

Compiling my own version where those three lines are commented out, does result in a dotnet where I can open well over 10k files (my test ran to 250K files before I stopped it).

It would be useful if that artificial limit did not apply to the released version of dotnet.

Reproduction Steps
var directory = "/tmp/openfiles/";
        if(Directory.Exists(directory)) Directory.Delete(directory, true);
        
        Directory.CreateDirectory(directory);

        var fhs = new List<FileStream>();
        for (int i = 0; i < 30000; i++)
        {
            var file = File.Open(directory + "/" + i + ".txt", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite);
            fhs.Add(file);
            
            if(i % 1000 == 0) Console.WriteLine($"File {i} opened");
        }
Expected behavior

Can open more than 10240 files.

Actual behavior

fails to open more than 10kish files.

Regression?

no

Known Workarounds

none

Configuration

.net 8.0.13

Other information

No response

Dominant language
C#
Stars
18.3k
Forks
5.6k
PR metrics pending

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/runtime

All issues in dotnet/runtime

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.