get2knowio / get2knowio/remo

OrbStack: `apt-get install` fails inside any running container (dpkg: Invalid cross-device link) — breaks runtime installs and E2E browser tests, silently

Open
#172 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2
Forks
0
Avg merge
15h 59m
Merged PRs (30d)
9

Description

## Summary

On an OrbStack host, **`apt-get install` fails for essentially any package inside any running container**. This is not font-, Playwright-, or devcontainer-specific — it reproduces with `debian:trixie` and one package:

```
$ docker run --rm debian:trixie sh -c 'apt-get update -qq; apt-get install -y jq'
Unpacking jq (1.7.1-6+deb13u3) ...
dpkg: error processing archive /var/cache/apt/archives/jq_1.7.1-6+deb13u3_arm64.deb (--unpack):
unable to install new version of './usr/share/doc/jq': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (1)
```

This is a third distinct consequence of the same nested-overlayfs platform limitation as #160 (build-time BuildKit) and #171 (Compose builds). Filing separately because the layer is different — this one is **runtime**, inside an already-built container, and no builder configuration fixes it.

## Cause

The container's root overlay is mounted with `redirect_dir=nofollow`:

```
/ / overlay rw,lowerdir=...snapshots/250/fs:...snapshots/239/fs,
upperdir=...snapshots/251/fs,workdir=...snapshots/251/work,
redirect_dir=nofollow,uuid=null
```

and the kernel module default agrees:

```
$ cat /sys/module/overlay/parameters/redirect_dir
N
```

dpkg unpacks each package's `./usr/share/doc//` and then `rename()`s the staged directory into place. `/usr/share/doc` exists only in a **lower** layer (it comes from the base image), so renaming a directory across layers requires `redirect_dir`. Without it overlayfs returns `EXDEV`, which surfaces as dpkg's "Invalid cross-device link". Because virtually every Debian package ships a `/usr/share/doc/` directory, virtually every package fails.

## Why builds are unaffected

The `remo-native` builder from #160 uses the **native** snapshotter, which copies layers instead of overlaying them, so dpkg's rename stays within one filesystem. The contrast is exact — same host, same package, same base image:

| | result |
|---|---|
| `docker buildx build --builder remo-native` (native snapshotter) | `APT_OK_AT_BUILD_TIME` |
| `docker run` (containerd overlayfs snapshotter) | `Invalid cross-device link` |

So #160's fix silently covers the build path only. Devcontainer Features that apt-install work fine — they run at build time. Anything that apt-installs at **runtime** fails.

## Impact

Any `postCreateCommand`, `postStartCommand`, or interactive `apt-get install` in a devcontainer on such a host fails. The concrete casualty found in practice:

```
npm exec -- playwright install --with-deps chromium
...
dpkg: error processing archive .../fonts-ipafont-gothic_00303-23_all.deb (--unpack):
unable to install new version of './usr/share/doc/fonts-ipafont-gothic':
Invalid cross-device link
Failed to install browsers
Error: Installation process exited with code: 100
```

**E2E browser tests therefore cannot run on an affected host at all.**

This one is nastier than #160/#171 because it fails *quietly*. A well-written post-create script treats a browser-install failure as non-fatal, so `devcontainer up` still reports success:

```
{"outcome":"success","containerId":"...","remoteUser":"node", ...}
```

while `~/.cache/ms-playwright` is empty. The host looks healthy, the devcontainer looks healthy, and the breakage only appears later when someone runs the E2E suite. Anyone checking "did the devcontainer come up?" gets a green answer.

## Possible remediations

**Stated as candidates — none of these were tested**, because each requires restarting the Docker daemon, which would have killed running devcontainers on the host under investigation. They are ordered by how promising they look.

1. **Switch containerd to the btrfs snapshotter.** An OrbStack machine's root is btrfs, so this is available and would avoid overlayfs entirely for runtime containers:
```
$ findmnt -no FSTYPE,SOURCE /
btrfs /dev/vdb1[/scon/containers//rootfs]
```
This mirrors what #160 did for builds (native snapshotter) at the runtime layer. Cost profile is unknown and should be measured — btrfs snapshots are cheap, so it may be much better than `native`.

2. **Disable the containerd snapshotter**, reverting to the legacy `overlay2` graph driver, via `/etc/docker/daemon.json`:
```json
{ "features": { "containerd-snapshotter": false } }
```
Worth testing but not obviously a fix — `overlay2` is still overlayfs, so it may fail identically unless it negotiates `redirect_dir` differently. Note #160 observed that Docker 29 making the containerd snapshotter the default is what moved these hosts onto the broken path.

3. **Enable `redirect_dir` on the overlay module** (`/sys/module/overlay/parameters/redirect_dir`, currently `N`). Addresses the mechanism most directly, but it is a kernel-module parameter on a nested/shared kernel, so it may simply be refused — and on OrbStack the kernel is not the machine's to configure.

There is currently **no `/etc/docker/daemon.json` at all** on the affected host, so any of these would be a new file and remo has a clean slate to write one.

If none prove workable, the fallback is the "warn loudly" path floated in #160: `remo configure` should say plainly that runtime `apt-get` does not work on OrbStack machines, because the failure mode is otherwise invisible until a test suite fails for unrelated-looking reasons.

## Reproduction

```bash
# fails
docker run --rm debian:trixie sh -c 'apt-get update -qq; apt-get install -y jq'

# succeeds on the same host, same package
docker buildx build --builder remo-native --no-cache - <<'EOF'
FROM debian:trixie
RUN apt-get update -qq && apt-get install -y jq && echo APT_OK_AT_BUILD_TIME
EOF
```

## Environment

remo CLI 4.3.2; OrbStack machine (arm64), kernel `7.0.11-orbstack-00360-gc9bc4d96ac70`; Docker 29.7.2, storage driver `overlayfs` / `io.containerd.snapshotter.v1`; no `daemon.json`; host root btrfs; `remo-native` builder present per #160.

Related: #160 (build-time BuildKit on the same limitation), #171 (Compose-based devcontainer builds, which also notes this failure in passing).

Contributor guide

Open the contributing guide

Research direction

Reproduce the runtime failure with the Debian container command and compare it with the successful remo-native build, then review related issues #160 and #171. Investigate the listed containerd, overlayfs, and daemon configuration candidates; done means identifying a tested runtime remediation or making remo configure warn clearly about the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.