apple / apple/container

[Bug]: completed exec state is retained in container-runtime-linux

Open
#2,057 1 comment 0 reactions 0 assignees View on GitHub
reliability
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### Description

In Apple Container 1.2.0 (`6e65319`), a long-lived
`container-runtime-linux` process retains host-side state for every completed
exec. Repeated sequential `container machine run ... /usr/bin/true` calls make
the runtime's live allocations and physical footprint grow approximately in
proportion to the completed command count. Stopping the machine reclaims the
state because it terminates the runtime helper, but a long-lived helper does
not plateau.

### Sanitized reproduction

1. Start one machine and keep the same `container-runtime-linux` PID and guest
boot for the entire test.
2. Record the helper's live malloc allocation count and physical footprint.
3. Run thousands of sequential, concurrency-one `/usr/bin/true` commands with
`container machine run`.
4. Allow an idle settle period and measure the same helper again.
5. Repeat another equal command batch.

The completed-command count predicts the retained allocation growth, and
pausing the command source stops that growth. A 100-command probe left 13,589
additional live allocations in the helper in one measured run.

### Source trace

The current exec path appears to leave four per-exec objects reachable:

- `RuntimeService.processes[id]`
- `RuntimeService.waiters[id]`
- `ExitMonitor.exitCallbacks[id]`
- `ExitMonitor.runningTasks[id]`

`RuntimeService.createProcess` inserts the process and waiter and registers an
exit callback. The exit callback releases the waiter, deletes the underlying
process, and marks the process stopped, but it does not remove the process or
waiter entries or call `ExitMonitor.stopTracking`. `ExitMonitor.track` also
does not remove its completed callback/task entries.

Simply deleting the entries in the callback is racy because the current
protocol does not declare whether an exec will later be waited or is detached.
An exec can exit before a normal caller sends `wait`, while a detached exec
will never send it.

### Expected behavior

Completed normal, detached, failed-start, canceled-client, signal, and shutdown
paths should release all per-exec host state and return to a stable allocation
baseline without requiring a machine restart. Completion ownership should be
declared before start (for example, atomic wait-and-reap versus reap-on-exit),
so cleanup does not depend on whether a late wait happens to arrive.

I am preparing a focused patch and deterministic normal/detached/failed-start
stress coverage against this upstream baseline.

Contributor guide

Open the contributing guide

Research direction

Start by reading RuntimeService.createProcess and ExitMonitor.track, then reproduce the sequential /usr/bin/true stress case while measuring the helper's allocations and physical footprint. Trace normal, detached, failed-start, canceled-client, signal, and shutdown completion paths; done means all per-exec host state is released and the allocation baseline stabilizes without restarting the machine.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, swift
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.