chmod failed when `mountInotify` is enabled
- Dominant language
- Go
- Stars
- 21.9k
- Forks
- 957
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 53
Description
# Overview
After enabling the `mountInotify` feature, files on virtiofs faied to chmod under specific conditions.
I tried some debugging but could not figure out the cause.
# Environment
* MacBook Pro(Apple M3)
* macOS 14.5(Sonoma)
```bash
$ limactl --version
limactl version 0.22.0
$ uname -v
Darwin Kernel Version 23.5.0: Wed May 1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031
```
# Steps to reproduce
## 1. Create a template file
`debug-inotify.yaml`
```
vmType: "vz"
networks:
- vzNAT: true
mountType: "virtiofs"
mountInotify: true
images:
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-arm64.img"
arch: "aarch64"
digest: "sha256:c841bac00925d3e6892d979798103a867931f255f28fefd9d5e07e3e22d0ef22"
mounts:
- location: "~"
writable: false
- location: "/tmp/lima"
writable: true
containerd:
system: false
user: false
```
## 2. Create a VM
```bash
$ limactl start --name=debug-inotify ./debug-inotify.yaml
$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ sudo apt -y install unzip
kuro@lima-debug-inotify:~$ exit
```
## 3. Download zip file(ex. terraform) and extract
```bash
$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ cd /tmp/lima/
kuro@lima-debug-inotify:/tmp/lima$ wget https://releases.hashicorp.com/terraform/1.9.0/terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ ls
terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ unzip terraform_1.9.0_darwin_arm64.zip
Archive: terraform_1.9.0_darwin_arm64.zip
inflating: LICENSE.txt
inflating: terraform fchmod (file attributes) error: Operation not permitted
(warning) cannot set modif./access times
Operation not permitted
kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro 4922 Jun 27 00:59 LICENSE.txt
-rw-r--r-- 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ chmod +x ./terraform
kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro 4922 Jun 27 00:59 LICENSE.txt
-rwxr-xr-x 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip
```
The error `fchmod (file attributes) error: Operation not permitted` occurred and the binary `terraform` does not have executable flag. However, after `chmod +x` manually, it have executable flag.
# Debug
## strace
```bash
kuro@lima-debug-inotify:/tmp/lima$ strace unzip terraform_1.9.0_darwin_arm64.zip
...[snip]
read(3, "\351\273\215\23\214\344\202\335\373y\302\366\326\300\241Q\216\356X\30\221\204\362\304\240\10\30\30\tK\360 "..., 8192) = 3818
write(4, "\2\270\374\t\365\343\362<\373:8\3335\360\306Q\376?\245j`coL\272\202\234\360\30r\374\247"..., 35024) = 35024
fchmod(4, 0100755) = -1 EPERM (Operation not permitted)
dup(2) = 5
fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(5, "fchmod (file attributes) error: "..., 56fchmod (file attributes) error: Operation not permitted
) = 56
close(5) = 0
...[snip]
```
`fchmod(4, 0100755) = -1 EPERM (Operation not permitted)` seems failed to chmod.
## Customize `guestagent`
I implemented three types of function `HandleInotify`.
https://github.com/lima-vm/lima/blob/38c62c5e815fdc93e6e88543be1eb9ef7699eaf5/pkg/guestagent/guestagent_linux.go#L337-L346
### a. Remove `os.Ctimes()`
This causes the same issue.
```go
func (a *agent) HandleInotify(event *api.Inotify) {
location := event.MountPath
if _, err := os.Stat(location); err == nil {
logrus.Infof("inotify handle. Event: %s", event)
}
}
```
### b. Remove everything
This does **NOT** cause problems.
```go
func (a *agent) HandleInotify(event *api.Inotify) {
logrus.Infof("inotify handle. Event: %s", event)
}
```
### c. Insert delay before `os.Stat()`
This does **NOT** cause problems. (100ms is bad, but 500ms or later is ok)
```go
func (a *agent) HandleInotify(event *api.Inotify) {
location := event.MountPath
go func(){
time.Sleep(500 * time.Millisecond)
if _, err := os.Stat(location); err == nil {
logrus.Infof("inotify handle. Event: %s", event)
}
}()
}
```
Contributor guide
Assessment
This issue has not been assessed yet.