Attempting to open >256 files results in `Operation not permitted`
- Ngôn ngữ chính
- Go
- Star
- 30.8k
- Fork
- 613
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Description
I encountered this while doing a `cargo build`, while `cargo` was attempting to build `openssl`. OpenSSL's build includes a `ar` command that archives quite a lot of `.o` files into one archive. This isn't particularly unusual.
However, that process fails:
```
make: Warning: File 'Makefile' has modification time 1.5 s in the future
make: warning: Clock skew detected. Your build may be incomplete.
ar: crypto/asn1/libcrypto-lib-d2i_pr.o: Operation not permitted
make[1]: *** [Makefile:4160: libcrypto.a] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:2134: build_libs] Error 2
```
If we attempt to replicate the failing `ar` command under an `strace`:
```
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_key.o", {st_mode=S_IFREG|0644, st_size=56776, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_key.o", O_RDONLY) = 251
fcntl(251, F_GETFD) = 0
fcntl(251, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_lib.o", {st_mode=S_IFREG|0644, st_size=44552, ...}, 0) = 0
brk(0x5568116c3000) = 0x5568116c3000
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_lib.o", O_RDONLY) = 252
fcntl(252, F_GETFD) = 0
fcntl(252, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_meth.o", {st_mode=S_IFREG|0644, st_size=25328, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_meth.o", O_RDONLY) = 253
fcntl(253, F_GETFD) = 0
fcntl(253, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_pmeth.o", {st_mode=S_IFREG|0644, st_size=73688, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_pmeth.o", O_RDONLY) = 254
fcntl(254, F_GETFD) = 0
fcntl(254, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_prn.o", {st_mode=S_IFREG|0644, st_size=9936, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_prn.o", O_RDONLY) = 255
fcntl(255, F_GETFD) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_rfc5114.o", {st_mode=S_IFREG|0644, st_size=12280, ...}, 0) = 0
openat(AT_FDCWD, "crypto/dh/libcrypto-lib-dh_rfc5114.o", O_RDONLY) = -1 EPERM (Operation not permitted)
write(2, "ar: crypto/dh/libcrypto-lib-dh_r"..., 66ar: crypto/dh/libcrypto-lib-dh_rfc5114.o: Operation not permitted
) = 66
```
We see that the prior successes end with the successful opening of file descriptor `255`, and the next `openat` fails, which is a suspect point to fail at. `ulimit` indicates that's not the problem, and all these files have the same owner/perms, so a real permission error is also not the problem.
We can replicate this quite simply by just attempting to `ar` 300 empty files together, and thus remove OpenSSL from the repro test:
1. Start a container; mount a host directory & cd into it. (This command does all that.) I'm trying to actually get the output of the build once it finishes, of course… 😉
```
docker run --rm -ti -w /src --mount type=bind,src=$(pwd),dst=/src rust:slim-bookworm
```
2. Attempt to do the `ar`:
```
# mkdir foo && cd foo && for i in `seq 1 300`; do touch $i; done; ar qc test.a *
ar: 56: Operation not permitted
```
N.b. that "56" ends up being the 256th file. The glob seems to pass the files in lexicographic order, so we open 1, 10, 100, 101, etc. first. `strace`:
```
# Grab strace:
apt-get update && apt-get install --yes --no-install-recommends strace
# strace:
strace ar qc test.a *
[snip]
openat(AT_FDCWD, "55", O_RDONLY) = 255
fcntl(255, F_GETFD) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
newfstatat(AT_FDCWD, "56", {st_mode=S_IFREG|0644, st_size=0, ...}, 0) = 0
openat(AT_FDCWD, "56", O_RDONLY) = -1 EPERM (Operation not permitted)
```
### Version
```
» colima version && limactl --version && qemu-img --version
colima version 0.5.6
git commit: ceef812c32ab74a49df9f270e048e5dced85f932
limactl version 0.18.0
qemu-img version 8.1.2
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
```
### Operating System
- [X] macOS Intel <= 13 (Ventura)
- [ ] macOS Intel >= 14 (Sonoma)
- [ ] Apple Silicon <= 13 (Ventura)
- [ ] Apple Silicon >= 14 (Sonoma)
- [ ] Linux
### Output of `colima status`
(`colima` started failing to start for me while filing this bug report. So, now I'm going to attempt to figure out what bug *that* is…)
Edit: Okay, working again, still bugged:
```
» colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/roy/.colima/default/docker.sock
```
### Reproduction Steps
Open >256 files in a single process inside the container.
### Expected behaviour
It works.
### Additional context
_No response_
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
The issue is about a file descriptor limit when opening >256 files inside a container. Look at the colima code that sets up the container environment, possibly in the VM configuration or mount handling. The strace output shows the failure at FD 255. Investigate how colima configures resource limits or file system mounts. Check if there's a known limit in the underlying lima or QEMU layer. Reproduce the bug with the provided steps to confirm the issue.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- docker, go, linux
- Lĩnh vực
- cli, operating-systems
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 35/100