CIME container fails on NCAR HPC systems (Derecho/Casper) due to rootless Podman configuration
- Dominant language
- Python
- Stars
- 174
- Forks
- 225
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 18
Description
## Summary
Testing of the CIME container on NCAR HPC systems revealed issues with rootless Podman configuration that prevent the container from running properly.
## Background
From discussion in #4960, @samsrabin attempted to run the CIME container on NCAR's Casper and Derecho systems and encountered several issues related to rootless Podman configuration.
## Issues Encountered
### 1. Login Node Behavior
- **Casper login nodes**: Container process is `Killed`
- **Casper compute nodes**: Container pulls blobs but returns to prompt without starting
- **Derecho login nodes**: Permission denied error during layer extraction
### 2. Rootless Podman Configuration Problem
The core issue is that NCAR HPC systems force rootless Podman, but user accounts lack subuid/subgid ranges in `/etc/subuid` and `/etc/subgid`. This causes:
```
ERRO[0000] cannot find UID/GID for user samrabin: no subuid ranges found for user "samrabin" in /etc/subuid - check rootless mode in man pages.
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user
```
With single-UID mapping, Podman cannot extract layers containing files not owned by root (e.g., `/home/ubuntu` in the Ubuntu base image), leading to:
```
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:...": ApplyLayer stdout: stderr: setting up pivot dir: mkdir /glade/u/home/samrabin/.local/share/containers/storage/vfs/dir/.../.pivot_root...: permission denied exit status 1
```
### 3. Storage Location Issue
The default Podman storage location (`~/.local/share/containers`) is on GLADE distributed storage, forcing use of the slow `vfs` driver.
## Proposed Workarounds
Two potential solutions were suggested in #4960:
1. **Ignore chown errors** during extraction:
```bash
podman --storage-opt vfs.ignore_chown_errors=true run ...
```
2. **Use local disk for image store** instead of GLADE:
```bash
podman --root /tmp/$USER/containers/storage --runroot /tmp/$USER/containers/run run ...
```
## Action Items
- [ ] Test the proposed workarounds on NCAR systems
- [ ] Update documentation with NCAR-specific instructions if workarounds are successful
- [ ] Consider making the container rootless-compatible to avoid these issues
- [ ] Document known limitations on HPC systems with restrictive rootless Podman configurations
## Related
- Original discussion: #4960
- Test command: `podman run -it --rm --hostname docker -e CIME_MODEL=cesm -v $PWD:/root/model -v ./storage:/root/storage -w /root/model/cime ghcr.io/esmci/cime:latest bash`
## Additional Context
The container runs successfully on standard workstations/laptops with properly configured rootless Podman. The issues are specific to the NCAR HPC environment's rootless Podman configuration.
Contributor guide
Research direction
Start with the test command in the issue and reproduce it on NCAR's Casper and Derecho systems. Test the proposed storage and chown workarounds, using discussion #4960 for context. Done means identifying which workarounds succeed and documenting NCAR-specific instructions and remaining rootless Podman limitations.
Written by the indexing model from the issue text.
Assessment
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100