cloudflare / cloudflare/containers
`/dev/shm` is mounted, then shadowed by a later devtmpfs mount on `/dev` — breaks POSIX shared memory for non-root images
- Dominant language
- TypeScript
- Stars
- 270
- Forks
- 42
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 4
Description
## Summary
On GA Containers, POSIX shared memory is unavailable to **non-root images**: `/dev/shm` does not exist in the visible filesystem, so glibc's `sem_open()` fails and Python's `multiprocessing` raises `FileNotFoundError`.
This does not look like an intentional limitation. The runtime **does** mount an shm tmpfs at `/dev/shm`, and then mounts a devtmpfs over `/dev`, shadowing it. The mount exists but is unreachable.
## Evidence
`/proc/mounts` from a running container (captured 2026-09-07):
```
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 ← shm IS mounted
devtmpfs /dev devtmpfs rw,relatime,size=6254124k,nr_inodes=1563531,mode=755 0 0 ← then /dev is over-mounted
```
The later `devtmpfs` on `/dev` shadows everything under the earlier `/dev` mount, including the `shm` mountpoint. `os.listdir("/dev")` contains no `shm`.
## Minimal reproduction
Any non-root image (ours is based on `python:3.12-slim` with `USER` set to a uid-1000 account):
```python
from multiprocessing import Pool
with Pool(2) as p:
p.map(abs, [-1, -2])
# FileNotFoundError: [Errno 2] No such file or directory
```
```python
import os
os.path.exists("/dev/shm") # False
```
## Expected vs actual
- **Expected:** `/dev/shm` is usable, or the limitation is documented.
- **Actual:** it is mounted but shadowed, and unrepairable from inside the guest. The visible `/dev` is root-owned `mode=755` and the workload runs unprivileged, so `mkdir /dev/shm` → `PermissionError`, and `mount -t tmpfs` → `must be superuser to use mount`.
## Why this is easy to miss
A **root** image can `mkdir /dev/shm` on the devtmpfs and never notice the problem, which may be why this survived GA. Only unprivileged images are affected.
## Impact
The failure mode is worse than a crash, because the most common path degrades silently:
- `multiprocessing.Pool` raises `FileNotFoundError` — an error that says nothing about shared memory and is very hard to diagnose from application code.
- **joblib detects the failure at import and silently falls back to serial.** `sklearn`'s `n_jobs=-1` then runs, returns correct results, and uses one core, with no warning reaching the user.
Measured on a `standard-4` instance (4 vCPU), scikit-learn `RandomForestClassifier`:
| | `n_jobs=1` | `n_jobs=-1` | speedup |
| --- | --- | --- | --- |
| Cloudflare Containers `standard-4` | 2.80 s | 2.71 s | **1.03x** |
| Local 12-core machine, same code | 1.96 s | 0.27 s | 7.26x |
Thread-based parallelism is unaffected (OpenBLAS/OpenMP correctly report 4 threads), so numpy/polars/duckdb work fine. The `multiprocessing`/`joblib`/`sklearn` family does not, which means a multi-vCPU instance type buys nothing for that entire class of workload.
Switching joblib to its threading backend is **not** an effective workaround — measured 0.90x (slower) for `RandomForestClassifier` and 1.03x for `ExtraTreesClassifier` on a 20k x 40 dataset.
## Suggested fix
Mount the shm tmpfs **after** the devtmpfs is mounted on `/dev`, so it is not shadowed. Alternatively, create `/dev/shm` with mode `1777` on the devtmpfs after it is mounted — `sem_open()` only needs a writable directory at that path.
The current 64 MB size is fine for semaphores and locks, though it is small for joblib's large-array memmapping; raising it would help heavier shared-memory workloads.
## Environment
- Cloudflare Containers, GA
- Instance types: `standard-3` and `standard-4` (both affected)
- Image base: `python:3.12-slim`, non-root (`USER` uid 1000)
- Docs checked for any mention of `/dev/shm`, shared memory, or process-isolation caveats (limits, platform details, changelog, known issues) — none found.
Contributor guide
Research direction
No repository file or test is named. Start by locating the container runtime's device and shared-memory mount setup, then reproduce the issue with the supplied non-root Python example and inspect /proc/mounts. Done means /dev/shm is visible and usable for POSIX shared memory without requiring guest-side privileges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- cloud, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100