OrbStack: `apt-get install` fails inside any running container (dpkg: Invalid cross-device link) — breaks runtime installs and E2E browser tests, silently
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 15h 59m
- Merged PRs (30d)
- 9
Description
## Summary
On an OrbStack host, **`apt-get install` fails for essentially any package inside any running container**. This is not font-, Playwright-, or devcontainer-specific — it reproduces with `debian:trixie` and one package:
```
$ docker run --rm debian:trixie sh -c 'apt-get update -qq; apt-get install -y jq'
Unpacking jq (1.7.1-6+deb13u3) ...
dpkg: error processing archive /var/cache/apt/archives/jq_1.7.1-6+deb13u3_arm64.deb (--unpack):
unable to install new version of './usr/share/doc/jq': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (1)
```
This is a third distinct consequence of the same nested-overlayfs platform limitation as #160 (build-time BuildKit) and #171 (Compose builds). Filing separately because the layer is different — this one is **runtime**, inside an already-built container, and no builder configuration fixes it.
## Cause
The container's root overlay is mounted with `redirect_dir=nofollow`:
```
/ / overlay rw,lowerdir=...snapshots/250/fs:...snapshots/239/fs,
upperdir=...snapshots/251/fs,workdir=...snapshots/251/work,
redirect_dir=nofollow,uuid=null
```
and the kernel module default agrees:
```
$ cat /sys/module/overlay/parameters/redirect_dir
N
```
dpkg unpacks each package's `./usr/share/doc//` and then `rename()`s the staged directory into place. `/usr/share/doc` exists only in a **lower** layer (it comes from the base image), so renaming a directory across layers requires `redirect_dir`. Without it overlayfs returns `EXDEV`, which surfaces as dpkg's "Invalid cross-device link". Because virtually every Debian package ships a `/usr/share/doc/` directory, virtually every package fails.
## Why builds are unaffected
The `remo-native` builder from #160 uses the **native** snapshotter, which copies layers instead of overlaying them, so dpkg's rename stays within one filesystem. The contrast is exact — same host, same package, same base image:
| | result |
|---|---|
| `docker buildx build --builder remo-native` (native snapshotter) | `APT_OK_AT_BUILD_TIME` |
| `docker run` (containerd overlayfs snapshotter) | `Invalid cross-device link` |
So #160's fix silently covers the build path only. Devcontainer Features that apt-install work fine — they run at build time. Anything that apt-installs at **runtime** fails.
## Impact
Any `postCreateCommand`, `postStartCommand`, or interactive `apt-get install` in a devcontainer on such a host fails. The concrete casualty found in practice:
```
npm exec -- playwright install --with-deps chromium
...
dpkg: error processing archive .../fonts-ipafont-gothic_00303-23_all.deb (--unpack):
unable to install new version of './usr/share/doc/fonts-ipafont-gothic':
Invalid cross-device link
Failed to install browsers
Error: Installation process exited with code: 100
```
**E2E browser tests therefore cannot run on an affected host at all.**
This one is nastier than #160/#171 because it fails *quietly*. A well-written post-create script treats a browser-install failure as non-fatal, so `devcontainer up` still reports success:
```
{"outcome":"success","containerId":"...","remoteUser":"node", ...}
```
while `~/.cache/ms-playwright` is empty. The host looks healthy, the devcontainer looks healthy, and the breakage only appears later when someone runs the E2E suite. Anyone checking "did the devcontainer come up?" gets a green answer.
## Possible remediations
**Stated as candidates — none of these were tested**, because each requires restarting the Docker daemon, which would have killed running devcontainers on the host under investigation. They are ordered by how promising they look.
1. **Switch containerd to the btrfs snapshotter.** An OrbStack machine's root is btrfs, so this is available and would avoid overlayfs entirely for runtime containers:
```
$ findmnt -no FSTYPE,SOURCE /
btrfs /dev/vdb1[/scon/containers//rootfs]
```
This mirrors what #160 did for builds (native snapshotter) at the runtime layer. Cost profile is unknown and should be measured — btrfs snapshots are cheap, so it may be much better than `native`.
2. **Disable the containerd snapshotter**, reverting to the legacy `overlay2` graph driver, via `/etc/docker/daemon.json`:
```json
{ "features": { "containerd-snapshotter": false } }
```
Worth testing but not obviously a fix — `overlay2` is still overlayfs, so it may fail identically unless it negotiates `redirect_dir` differently. Note #160 observed that Docker 29 making the containerd snapshotter the default is what moved these hosts onto the broken path.
3. **Enable `redirect_dir` on the overlay module** (`/sys/module/overlay/parameters/redirect_dir`, currently `N`). Addresses the mechanism most directly, but it is a kernel-module parameter on a nested/shared kernel, so it may simply be refused — and on OrbStack the kernel is not the machine's to configure.
There is currently **no `/etc/docker/daemon.json` at all** on the affected host, so any of these would be a new file and remo has a clean slate to write one.
If none prove workable, the fallback is the "warn loudly" path floated in #160: `remo configure` should say plainly that runtime `apt-get` does not work on OrbStack machines, because the failure mode is otherwise invisible until a test suite fails for unrelated-looking reasons.
## Reproduction
```bash
# fails
docker run --rm debian:trixie sh -c 'apt-get update -qq; apt-get install -y jq'
# succeeds on the same host, same package
docker buildx build --builder remo-native --no-cache - <<'EOF'
FROM debian:trixie
RUN apt-get update -qq && apt-get install -y jq && echo APT_OK_AT_BUILD_TIME
EOF
```
## Environment
remo CLI 4.3.2; OrbStack machine (arm64), kernel `7.0.11-orbstack-00360-gc9bc4d96ac70`; Docker 29.7.2, storage driver `overlayfs` / `io.containerd.snapshotter.v1`; no `daemon.json`; host root btrfs; `remo-native` builder present per #160.
Related: #160 (build-time BuildKit on the same limitation), #171 (Compose-based devcontainer builds, which also notes this failure in passing).
Contributor guide
Research direction
Reproduce the runtime failure with the Debian container command and compare it with the successful remo-native build, then review related issues #160 and #171. Investigate the listed containerd, overlayfs, and daemon configuration candidates; done means identifying a tested runtime remediation or making remo configure warn clearly about the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100