Too many open files on Mac OS with JDK 11 and mvnd 0.8.2
- Dominant language
- Java
- Stars
- 3.5k
- Forks
- 250
- Avg merge
- 16h 22m
- Merged PRs (30d)
- 36
Description
We have a very large project and in our day to day development also when we upgraded from JDK8 to JDK11, when running regular maven builds we used to get `Too many open files` error. We circumvented this in 2 ways.
1. On mac , we created a max.limitfiles.plist file with a huge 10 million value.
2. We passed `-XX:-MaxFDLimit` under `.mvn/jvm.config`
Together after these 2 running `mvnw clean install -DskipTests -T16` worked fine as opposed to giving `Too many open files`.
The system is Apple M1 Max with JDK 11 and maven 3.8
We wanted to try maven daemon and we faced the same `Too many open files` issue when building our project. Doing lsof shows that it happens around 10k files ( since that is the JVM limit on Mac OS ) which is why `-XX:-MaxFDLimit` was needed.
We tried to create `.mvn/mvnd.properties` and put `mvnd.jvmArgs=-XX:-MaxFDLimit` however we still get this error.
On the shell we see that `ulimit -a` is set to the 10 million file value.
We can also see in the command that runs the `-Dmvnd.jvmArgs` having the above value.
Not sure if there is any steps that I could use to debug this issue.
Running `ps` on the maven daemon process
```
/nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3/bin/java -classpath /nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-common-0.8.2.jar:/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-agent-0.8.2.jar -javaagent:/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-agent-0.8.2.jar -XX:-MaxFDLimit -Dmvnd.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd -Dmaven.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn -Dmaven.conf=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/conf -Dmvnd.java.home=/nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3 -Dlogback.configurationFile=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/conf/logback.xml -Dmvnd.id=dc67aa8c -Dmvnd.daemonStorage=/Users/mkodnani/.m2/mvnd/registry/0.8.2 -Dmvnd.registry=/Users/mkodnani/.m2/mvnd/registry/0.8.2/registry.bin -Dmvnd.socketFamily=inet -Dmvnd.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd -Djdk.java.options=--add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-opens java.base/sun.nio.fs=ALL-UNNAMED -Dmvnd.noDaemon=false -Dmvnd.debug=false -Dmvnd.idleTimeout=3h -Dmvnd.keepAlive=100ms -Dmvnd.extClasspath= -Dmvnd.coreExtensions=com.zeus:zeus-extension:1.0.9 -Dmvnd.jvmArgs=-XX:-MaxFDLimit -Dmvnd.enableAssertions=false -Dmvnd.expirationCheckDelay=10s -Dmvnd.duplicateDaemonGracePeriod=10s -Dmvnd.socketFamily=inet org.mvndaemon.mvnd.common.MavenDaemon
```
you can see that `-Dmvnd.jvmArgs=-XX:-MaxFDLimit` is read from `.mvn/mvnd.properties` file.
Version:
```
❯ mvnd -version
mvnd 0.8.2 darwin-aarch64 native client (2bba2d6a4d3a5012ddf9f1f42a22784cad4011e3)
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Java version: 11.0.11, vendor: Azul Systems, Inc., runtime: /nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
```
Ulimit on the shell:
```
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) unlimited
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 10000000
Maximum stack size (kB, -s) 8176
Maximum amount of CPU time in seconds (seconds, -t) unlimited
Maximum number of processes available to current user (-u) 16000
Maximum amount of virtual memory available to each process (kB, -v) unlimited
```
```
> launchctl limit maxfiles
maxfiles 10000000 10000000
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the JVM arguments shown by `ps` with `.mvn/mvnd.properties` and `.mvn/jvm.config`, then inspect the reported `ulimit -a` and `launchctl limit maxfiles` values while reproducing the build with `mvnd`. Done means the large project can run its documented mvnd build on macOS with JDK 11 without the `Too many open files` failure, or the issue identifies the remaining limit and a reproducible diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, macos
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100