lima-vm / lima-vm/lima

store/disk: stale in_use_by lock after host crash prevents instance from restarting

Open
#4,929 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
21.9k
Forks
957
Avg merge
2d 6h
Merged PRs (30d)
53

Description

### Description

When a Lima instance is running and the host machine crashes (power loss, kernel panic, hard reboot), the `in_use_by` symlink at `~/.lima/_disks//in_use_by` is never cleaned up. On the next `limactl start`, the disk attach code sees the symlink and fails — even though no process actually holds the disk — leaving the instance permanently unbootable without manual intervention.

### Reproduction

Start an instance that uses an `additionalDisk`, then simulate a crash:

```bash
# kill the hostagent without giving it a chance to clean up (simulates crash)
kill -9 $(pgrep -f "lima-hostagent.*")

limactl start
# → fatal: failed to run attach disk "", in use by instance ""
```

The stale lock remains at `~/.lima/_disks//in_use_by` and must be removed manually:
```bash
rm ~/.lima/_disks//in_use_by
```

### Root Cause

`pkg/store/disk.go` implements locking as a plain symlink with no liveness information:

```go
func (d *Disk) Lock(instanceDir string) error {
inUseBy := filepath.Join(d.Dir, filenames.InUseBy)
return os.Symlink(instanceDir, inUseBy)
}

func (d *Disk) Unlock() error {
inUseBy := filepath.Join(d.Dir, filenames.InUseBy)
return os.Remove(inUseBy)
}
```

`Unlock()` is only called from a `cleanUp` callback in `pkg/hostagent/hostagent.go`. On a crash, `cleanUp` never runs.

Both the VZ driver (`pkg/driver/vz/vm_darwin.go`) and krunkit driver (`pkg/driver/krunkit/krunkit_darwin_arm64.go`) check `disk.Instance != ""` and immediately fail — there is no attempt to verify whether the locking process is still alive.

**Note:** `LockForInstance()` was added in a recent commit and the VZ driver updated to use it, which does auto-recover for same-instance restart. However, krunkit has not been updated and still uses the raw check. Neither approach verifies process liveness — they rely on exact path matching which is fragile under symlink or path normalization differences.

### Proposed Fix

Lima already writes `ha.pid` (`filenames.HostAgentPID`) to the instance directory on every start. This is the natural liveness signal.

In `LockForInstance` (and wherever `disk.Lock` is called directly), before rejecting a lock as "in use", verify the locking process is still alive by embedding the PID in the lock:

```go
// lock file content (JSON) — replaces or augments the plain symlink
type diskLock struct {
Instance string `json:"instance"`
InstanceDir string `json:"instanceDir"`
PID int `json:"pid"`
}
```

Attach logic:

```
lock exists?
→ read PID from lock
→ kill(pid, 0): alive? → legitimately in use, fail as today
→ kill(pid, 0): ESRCH? → stale lock (process is dead), remove and proceed
```

This is the same pattern used by PostgreSQL (`postmaster.pid`), flock-based daemons, and most system services for crash recovery. Since `ha.pid` is already written to the instance directory, Lima has all the information needed — it just needs to be consulted at lock-acquire time.

**Files to change:**
- `pkg/store/disk.go` — embed PID in lock, add stale detection to `LockForInstance`
- `pkg/driver/krunkit/krunkit_darwin_arm64.go` — replace raw `disk.Instance != ""` check with `LockForInstance`
- `pkg/store/disk_test.go` — add test for stale PID detection

### Environment

- Lima: v2.0.3 (via Colima v0.10.0 on macOS Darwin 24.6.0, Apple Silicon arm64)
- VM type: `vz`
- Host OS: macOS 15

### Impact

Any user whose machine crashes while a Lima-based instance is running is left with an instance that cannot start. The only recovery today is manual symlink deletion with no hint in the error message about how to fix it. This affects all Colima, Rancher Desktop, and Finch users using `additionalDisks`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.