runsc: no way to keep /tmp on the rootfs when the image's /tmp is empty (EXDEV vs Docker)
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
## Description
When the OCI spec has no mount at `/tmp` and the rootfs `/tmp` is **empty**, runsc mounts an internal tmpfs over it. That puts `/tmp` on a different device from the rootfs, so the common tempfile-then-`rename(2)` (or `link(2)`) pattern fails with `EXDEV` — where the same workload under runc/Docker succeeds, because Docker leaves `/tmp` as plain rootfs.
There appears to be no way to opt out: the tmpfs is skipped only when the rootfs `/tmp` is non-empty, and a spec-level bind mount of the image's own `tmp` directory doesn't help (it becomes a separate gofer mount — still a different device than an overlay-backed rootfs).
## Steps to reproduce
Any rootfs with an empty `/tmp` (repro verified with an extracted `ubuntu:24.04`):
```json
{
"ociVersion": "1.0.0",
"process": {
"user": {"uid": 0, "gid": 0},
"args": ["/bin/sh", "-c", "grep \" /tmp \" /proc/mounts || echo no-tmp-mount; echo x > /tmp/f; ln /tmp/f /f2 && echo LINK-OK || echo LINK-FAILED"],
"cwd": "/",
"env": ["PATH=/usr/bin:/bin"]
},
"root": {"path": "", "readonly": false},
"mounts": [{"destination": "/proc", "type": "proc", "source": "proc"}],
"linux": {"namespaces": [{"type": "pid"}, {"type": "mount"}]}
}
```
```console
# rootfs /tmp empty:
$ runsc --rootless --ignore-cgroups --network=none --overlay2=root:dir=/tmp/ov run --bundle . demo
none /tmp tmpfs rw,mode=01777 0 0
ln: failed to create hard link '/f2' => '/tmp/f': Invalid cross-device link
LINK-FAILED
# identical spec, after: touch /tmp/.keep
$ runsc --rootless --ignore-cgroups --network=none --overlay2=root:dir=/tmp/ov2 run --bundle . demo2
no-tmp-mount
LINK-OK
```
## Impact
Found running Terminal-Bench under Ray Sandbox (ray-project/ray#65744): tasks using Python's `tempfile` + `os.rename` into the working tree fail with `OSError: [Errno 18] Invalid cross-device link` on gVisor while passing under Docker. Our current workaround is seeding extracted image caches with a placeholder file (`/tmp/.ray-sandbox-keep`) so the emptiness check keeps `/tmp` on the rootfs — which pollutes shared image caches and leaks an artifact into sandboxes unless cleaned up post-boot.
## Feature request
A way to keep `/tmp` on the rootfs regardless of whether it's empty — e.g. a flag like `--tmp-mount=auto|tmpfs|rootfs` (defaulting to today's `auto`), or honoring an annotation in the spec. That would give integrators Docker-parity `/tmp` semantics without marker-file workarounds.
## runsc version
```
spec: 1.2.1
```
Contributor guide
Research direction
Start at the runsc mount setup that decides whether an empty rootfs /tmp receives an internal tmpfs, then reproduce the provided OCI bundle with the two rootfs variants. Done means an explicit configuration or annotation can keep /tmp on the rootfs and the shown hard-link or tempfile-then-rename workload succeeds without the marker-file workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- devops, operating-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100