lima-vm / lima-vm/lima

lima-guestagent sustains ~0.75 of a CPU core on an idle guest (2.1.1, colima/vz, aarch64)

Open
#5,473 0 comments 0 reactions 0 assignees View on GitHub
area/performance
Dominant language
Go
Stars
21.9k
Forks
957
Avg merge
2d 6h
Merged PRs (30d)
53

Description

### Summary

`lima-guestagent` 2.1.1 sustains **~0.75 of a CPU core** on a guest that is otherwise idle from a port-forwarding point of view. Over 47 ten-minute windows spanning seven hours it averaged **0.737 cores** (sd 0.152, range 0.455–0.997) — about 6% of a 12-vCPU guest, permanently.

**96.5% of it is system time** (utime 1,620,785 jiffies vs stime 44,319,479). Sampling `/proc//task/*/syscall` 400 times across 19 threads (7,600 samples): **82% futex, 8.6% epoll_pwait, 3.5% nanosleep, 5.3% userspace**. 5.4% of thread samples are runnable, which independently reproduces ~1 core.

### Environment

| | |
|---|---|
| lima-guestagent | 2.1.1 |
| driver | colima on macOS (Apple silicon), `vz` |
| guest | 12 vCPU, aarch64 |
| workload | ~38 containers, ~90 `bind(2)`/s steady state |
| `portForwarder` | `ssh` |

### What it is not — each ruled out by a controlled experiment

**Not `/proc/net/tcp` polling.** `/proc//io` over 10 s shows 50 reads and 38 writes (~9/s) and `read_bytes` does not move. `sendmsg`/`recvmsg` appear **zero** times in the 7,600 syscall samples, so it is not a netlink `sock_diag` dump either.

**Not the eBPF event rate.** The agent holds a 13-instruction `BPF_PROG_TYPE_TRACEPOINT`, a 1 MiB ringbuf and a perf link on `sys_exit_bind` (PR #3067). Injecting 1,452,954 `bind()` calls in 5 s moved `run_cnt` by 1,453,438 — a 1:1 match, so the tracepoint counts what it looks like it counts. But a controlled **22×** increase in the event rate (90/s → 2,016/s) moved CPU from **1.020 to 0.998 cores**. The BPF program's own cost is 1,913 runs in 20 s = 25.9 ms, **0.13% of a core**.

> Note for anyone reproducing: `run_cnt` and `run_time_ns` in a BPF fdinfo read **0** unless `kernel.bpf_stats_enabled=1`. They would otherwise "prove" the program never runs.

**Not memory.** `VmRSS` is 40,380 kB, byte for byte, both freshly started and after millions of events.

**Not the size of the port set.** 2,295,702 binds to one fixed port cost the same as 2,155,640 binds across ~28,000 ephemeral ports.

**Not fixable by restarting.** Six 60-second windows after a restart: 0.965 mean, against 0.927 as-found — **+4.1%, Welch t = 1.44**. (A shorter earlier look suggested a restart bought ~0.35 cores; a controlled repeat says it buys nothing.)

**Not `GOMAXPROCS`.** `GOMAXPROCS=1` (threads 18 → 8) measured 0.668 against 0.965 unset, which looks like −31% at t = −5.39. The **reversal arm refutes it**: reverting to the shipped configuration gave 0.756, not a climb back (t = 1.21), and a second `GOMAXPROCS=1` arm gave 0.760. The reverted arm also differs from the identically-configured earlier arm by 0.209 cores at **t = −4.16**, and the last arm decays from 0.967 to 0.658 across six minutes on its own. **The process's own drift is larger than any treatment we could apply**, which is itself part of this report.

### Questions

1. **Is ~0.75 of a core expected for an idle guestagent?** If not, a profile of where the futex traffic comes from would settle it — we cannot profile the Go runtime from outside the process.
2. **Is there a documented way to disable the eBPF port detector** and fall back to the 3-second polling it replaced? `lima-guestagent daemon --help` exposes only `--tick`, which is the polling interval rather than a gate on the eBPF path, and neither the docs nor #3067 name a flag or environment variable.
3. **Would a static `portForwards:` list bypass the detector?** Untested here, because disabling port forwarding blind would take down services that depend on it.

### Reproduction

A guest with a few dozen containers and a modest `bind(2)` rate; watch `ps -eo pid,pcpu,etimes,comm --sort=-pcpu`. **Measure over hours, not minutes** — see the drift note above. Short windows on this process produced three confident wrong answers in our own testing before a reversal arm caught them.

Contributor guide

Open the contributing guide

Research direction

Start with the lima-guestagent daemon entry point and PR #3067, then inspect the eBPF port detector and profile the reported futex traffic over hour-long reproduction windows. Check `lima-guestagent daemon --help` and the documentation for a detector-disable option or static `portForwards:` behavior. Done means explaining whether the CPU use is expected and identifying a documented workaround or a specific fix path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.