Attempting to open >256 files results in `Operation not permitted`
- Lenguaje dominante
- Go
- Estrellas
- 30.8k
- Forks
- 613
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
### Description
I encountered this while doing a `cargo build`, while `cargo` was attempting to build `openssl`. OpenSSL's build includes a `ar` command that archives quite a lot of `.o` files into one archive. This isn't particularly unusual.
However, that process fails:
```
make: Warning: File 'Makefile' has modification time 1.5 s in the future
make: warning: Clock skew detected. Your build may be incomplete.
ar: crypto/asn1/libcrypto-lib-d2i_pr.o: Operation not permitted
make[1]: *** [Makefile:4160: libcrypto.a] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:2134: build_libs] Error 2
```
If we attempt to replicate the failing `ar` command under an `strace`:
```
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_key.o", {st_mode=S_IFREG|0644, st_size=56776, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_key.o", O_RDONLY) = 251
fcntl(251, F_GETFD) = 0
fcntl(251, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_lib.o", {st_mode=S_IFREG|0644, st_size=44552, ...}, 0) = 0
brk(0x5568116c3000) = 0x5568116c3000
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_lib.o", O_RDONLY) = 252
fcntl(252, F_GETFD) = 0
fcntl(252, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_meth.o", {st_mode=S_IFREG|0644, st_size=25328, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_meth.o", O_RDONLY) = 253
fcntl(253, F_GETFD) = 0
fcntl(253, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_pmeth.o", {st_mode=S_IFREG|0644, st_size=73688, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_pmeth.o", O_RDONLY) = 254
fcntl(254, F_GETFD) = 0
fcntl(254, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_prn.o", {st_mode=S_IFREG|0644, st_size=9936, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_prn.o", O_RDONLY) = 255
fcntl(255, F_GETFD) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_rfc5114.o", {st_mode=S_IFREG|0644, st_size=12280, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_rfc5114.o", O_RDONLY) = -1 EPERM (Operation not permitted)
write(2, "ar: crypto/dh/libcrypto-lib-dh_r"..., 66ar: crypto/dh/libcrypto-lib-dh_rfc5114.o: Operation not permitted
) = 66
```
We see that the prior successes end with the successful opening of file descriptor `255`, and the next `openat` fails, which is a suspect point to fail at. `ulimit` indicates that's not the problem, and all these files have the same owner/perms, so a real permission error is also not the problem.
We can replicate this quite simply by just attempting to `ar` 300 empty files together, and thus remove OpenSSL from the repro test:
1. Start a container; mount a host directory & cd into it. (This command does all that.) I'm trying to actually get the output of the build once it finishes, of course… 😉
```
docker run --rm -ti -w /src --mount type=bind,src=$(pwd),dst=/src rust:slim-bookworm
```
2. Attempt to do the `ar`:
```
# mkdir foo && cd foo && for i in `seq 1 300`; do touch $i; done; ar qc test.a *
ar: 56: Operation not permitted
```
N.b. that "56" ends up being the 256th file. The glob seems to pass the files in lexicographic order, so we open 1, 10, 100, 101, etc. first. `strace`:
```
# Grab strace:
apt-get update && apt-get install --yes --no-install-recommends strace
# strace:
strace ar qc test.a *
[snip]
openat(AT_FDCWD, "55", O_RDONLY) = 255
fcntl(255, F_GETFD) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "56", {st_mode=S_IFREG|0644, st_size=0, ...}, 0) = 0
openat(AT_FDCWD, "56", O_RDONLY) = -1 EPERM (Operation not permitted)
```
### Version
```
» colima version && limactl --version && qemu-img --version
colima version 0.5.6
git commit: ceef812c32ab74a49df9f270e048e5dced85f932
limactl version 0.18.0
qemu-img version 8.1.2
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
```
### Operating System
- [X] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [ ] Apple Silicon <= 13 (Ventura)
- [ ] Apple Silicon >= 14 (Sonoma)
- [ ] Linux
### Output of `colima status`
(`colima` started failing to start for me while filing this bug report. So, now I'm going to attempt to figure out what bug *that* is…)
Edit: Okay, working again, still bugged:
```
» colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/roy/.colima/default/docker.sock
```
### Reproduction Steps
Open >256 files in a single process inside the container.
### Expected behaviour
It works.
### Additional context
_No response_
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
The issue is about a file descriptor limit when opening >256 files inside a container. Look at the colima code that sets up the container environment, possibly in the VM configuration or mount handling. The strace output shows the failure at FD 255. Investigate how colima configures resource limits or file system mounts. Check if there's a known limit in the underlying lima or QEMU layer. Reproduce the bug with the provided steps to confirm the issue.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- docker, go, linux
- Área
- cli, operating-systems
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bien especificado
- Aptitud para principiantes
- 35/100