bug: containerized Podman gateway broken on Fedora 44 (SELinux, UID mapping, port binding, state path)

Agent Diagnostic

Loaded the debug-openshell-cluster skill and investigated a live Fedora 44 host running the gateway as a rootless Podman container. Both the gateway and sandbox containers had exited ~21 hours prior.

Skills loaded: debug-openshell-cluster

Investigation steps and findings:

1. openshell status → Connection refused on http://127.0.0.1:8080 — gateway container had exited (code 0, clean shutdown signal)
2. podman logs openshell-gateway → Sandbox containers were immediately exiting with code 1; gateway log showed sandbox JWT auth not configured and a subuid/subgid warning
3. Inspected sandbox env → OPENSHELL_ENDPOINT=http://host.containers.internal:8080
4. Inspected gateway container → port mapped to 127.0.0.1:8080 only; sandbox on openshell bridge (10.89.1.0/24) resolves host.containers.internal to 10.89.1.1 — port 8080 not listening there → Connection refused
5. Recreated gateway with -p 0.0.0.0:8080:8080 → exited with Permission denied on Podman socket
6. Gateway image now runs as UID 1000:1000; without --userns=keep-id, container UID 1000 maps to host UID 100999, not socket owner tt (1000)
7. Added --userns=keep-id → DB volume files were owned by host UID 100999 (written by old container without keep-id) → unable to open database file
8. Fixed with podman unshare chown -R 0:0 on volume data → gateway started but hit SELinux AVC denial: container_t denied write on user_tmp_t sock_file
9. Added --security-opt label=disable → gateway reached Podman driver init; subuid warning fired (false positive — the check runs inside the container which doesn't have host's /etc/subuid mounted, but the host does have valid entries)
10. Gateway started but sandbox creation failed: "create sandbox token directory failed" — gateway tries to write to /app/.local/state/openshell/podman-sandbox-tokens/ inside container; that path is not writable by UID 1000 in the image
11. Set XDG_STATE_HOME=/run/user/1000/openshell-state with a mirrored bind mount (-v /run/user/1000/openshell-state:/run/user/1000/openshell-state) → token file written but Podman API returned 500: statfs ... sandbox.jwt: no such file or directory; gateway passes the container-internal path to the Podman API, which looks for it on the host — paths diverge unless source == destination in the bind mount
12. Used mirrored mount (host path == container path) → gateway wrote token, Podman found it, sandbox came up Ready
13. Also required: generating an Ed25519 JWT signing key pair and TOML config ([openshell.gateway.gateway_jwt] + [openshell.gateway.auth] allow_unauthenticated_users = true) — neither is documented for the Podman-in-container case

Description

I was following the official documentation at https://docs.nvidia.com/openshell/home. Happy to contribute improvements so that the next people using Fedora or SELinux won't run into these.

Actual behavior: Containerized Podman gateway fails to start or create sandboxes on Fedora 44 due to a cascade of issues: wrong port binding, UID/SELinux mismatches between container and host, missing sandbox JWT configuration, and a state-directory path mismatch between what the gateway writes and what the Podman API can find on the host.

Expected behavior: The documented podman run invocation for the gateway works out of the box on Fedora 44 with rootless Podman and SELinux enforcing, or the documentation covers each required flag and config step.

Reproduction Steps

1. Fedora 44 host, rootless Podman, SELinux enforcing
2. Run the gateway container as documented (or as shown in issue logs) with -p 127.0.0.1:8080:8080 and no --userns or SELinux flags
3. Attempt openshell sandbox create

Full working invocation (reached after debugging):

# Generate JWT signing keys (one-time)
mkdir -p ~/.config/openshell/jwt
openssl genpkey -algorithm ed25519 -out ~/.config/openshell/jwt/signing.pem
openssl pkey -in ~/.config/openshell/jwt/signing.pem -pubout -out ~/.config/openshell/jwt/public.pem
uuidgen > ~/.config/openshell/jwt/kid

# TOML config
cat > ~/.config/openshell/gateway.toml <<'TOML'
[openshell.gateway.auth]
allow_unauthenticated_users = true

[openshell.gateway.gateway_jwt]
signing_key_path = "/etc/openshell/jwt/signing.pem"
public_key_path  = "/etc/openshell/jwt/public.pem"
kid_path         = "/etc/openshell/jwt/kid"
gateway_id       = "openshell"
TOML

# State dir must be mirrored (same path on host and in container)
mkdir -p /run/user/1000/openshell-state

podman run -d --name openshell-gateway \
  --userns=keep-id \
  --security-opt label=disable \
  -p 0.0.0.0:8080:8080 \
  -v openshell-state:/var/openshell \
  -v /run/user/1000/podman/podman.sock:/var/run/podman.sock \
  -v ~/.config/openshell/jwt:/etc/openshell/jwt:ro \
  -v ~/.config/openshell/gateway.toml:/etc/openshell/gateway.toml:ro \
  -v /run/user/1000/openshell-state:/run/user/1000/openshell-state \
  -e OPENSHELL_DRIVERS=podman \
  -e OPENSHELL_PODMAN_SOCKET=/var/run/podman.sock \
  -e OPENSHELL_DB_URL=sqlite:/var/openshell/openshell.db \
  -e OPENSHELL_DISABLE_TLS=true \
  -e OPENSHELL_GATEWAY_CONFIG=/etc/openshell/gateway.toml \
  -e XDG_STATE_HOME=/run/user/1000/openshell-state \
  ghcr.io/nvidia/openshell/gateway:latest \
  --log-level debug --bind-address 0.0.0.0 --port 8080

Root Causes

Environment

• OS: Fedora 44
• Podman: rootless, SELinux enforcing
• OpenShell: 0.0.59
• Gateway deployment: containerized (podman run ghcr.io/nvidia/openshell/gateway:latest)

Logs

Gateway log (original failure — sandbox connection refused):

WARN openshell_server::compute: Sandbox failed to become ready sandbox_name=national-mammal reason=ContainerExited Container exited with code 1

Sandbox log (original failure):

Error: × Policy fetch failed after 5 attempts: failed to connect to OpenShell server

SELinux AVC denial (during debugging):

AVC avc: denied { write } for comm="openshell-gatew" name="podman.sock" dev="tmpfs"
  scontext=system_u:system_r:container_t:s0:c487,c938
  tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0

State path error (during debugging):

Error: × status: Internal, message: "create sandbox failed: podman API error (500):
  statfs /run/user/1000/openshell-state/openshell/podman-sandbox-tokens/.../sandbox.jwt:
  no such file or directory"