Bootc container lint fails on ARM64 containers via QEMU user-mode emulation (errno 38)

**TL;DR:** Some `bootc container lint` checks crash when running inside QEMU ARM64 user-mode emulation due to an unimplemented syscall (`set_robust_list`). A patch is available in QEMU to stub this syscall to success. As a workaround, users can skip the failing lints, apply the QEMU patch, or Bootc could be updated to gracefully handle ENOSYS.

**Context**: I discovered this issue while enabling RHEL Bootc to run on ARM-based SmartNICs (DPUs). Happy to make a PR with a fix, thanks for making Bootc rock-solid across architectures!
# 1. Description

When running `bootc container lint` inside an **ARM64 container on an x86_64 host** using QEMU user-mode emulation (e.g., via Podman's `--arch arm64` and binfmt\_misc), **only the `var-tmpfiles` and `utf8` lint checks fail**. They return:

```text
ERROR Linting: Unexpected runtime error running lint var-tmpfiles: I/O error: Function not implemented (os error 38)
```

However, **all other lint checks succeed**. This error only occurs when using QEMU user-mode emulation. It does not occur when:
* On native ARM64 (hardware or KVM)
* On native x86\_64
* When skipping the failing lints

---

# 2. Environment

| Key | Value |
| ----------------- | ---------------------------------------------------------------------------------- |
| Host Architecture | `x86_64` |
| Container Target | `aarch64` (ARM64) |
| Distro/Tag | `registry.redhat.io/rhel9/rhel-bootc:9.6` and `quay.io/fedora/fedora-bootc:latest` |
| Container Engine | `podman` with `--arch arm64` |
| QEMU Mode | `qemu-aarch64-static` via `binfmt_misc` |
| bootc Version | `1.1.6` (RHEL 9.6) and `1.4.0` (Fedora BootC latest) |

---

# 3. Reproduction Steps

### Using Fedora Bootc:

```bash
podman run --rm -it --arch arm64 \
quay.io/fedora/fedora-bootc:latest \
/bin/bash -c 'bootc container lint'
```

### Output:

```text
ERROR Linting: Unexpected runtime error running lint var-tmpfiles: I/O error: Function not implemented (os error 38)
```

Skipping `var-tmpfiles`:

```bash
bootc container lint --skip var-tmpfiles
```

Returns:

```text
ERROR Linting: Unexpected runtime error running lint utf8: Function not implemented (os error 38)
```

Skipping both:

```bash
bootc container lint --skip var-tmpfiles --skip utf8
```

Returns:

```text
Checks passed: 10
Checks skipped: 3
```

Reproducible across both RHEL and Fedora Bootc containers.

---

# 4. Logs / Evidence

QEMU trace:

```text
set_robust_list(...) = -1 errno=38 (Function not implemented)
Unsupported syscall: 293
ERROR Linting: Unexpected runtime error running lint var-tmpfiles: I/O error: Function not implemented (os error 38)
```

---

# 5. Root Cause

* On ARM64, syscall **293** is [`set_robust_list(2)`](https://man7.org/linux/man-pages/man2/set_robust_list.2.html), which is unimplemented in QEMU user-mode and returns ENOSYS (errno 38).
* glibc **calls `set_robust_list` during thread initialization**, which silently fails
* Bootc lints that trigger deeper metadata inspection (e.g., via `cap_std`) propagate the error

> The nature of the kernel ABI for `get/set_robust_list` means we cannot implement them in QEMU.
> [QEMU-devel mailing list, July 2025](https://lists.nongnu.org/archive/html/qemu-devel/2025-07/msg01553.html)

---

# 6. Possible reasons why only `var-tmpfiles` and `utf8` fail

| Lint | Explanation |
| ------------ | --------------------------------------------------------------------------------------------------- |
| var-tmpfiles | Traverses `/var` with metadata inspection via `cap_std::fs`, triggering the unimplemented syscall |
| utf8 | Resolves symlinks and checks file content encoding recursively, again triggering thread-related I/O |
| Others | Don't hit the same metadata paths or syscall paths so they pass |

---

# 7. Workarounds

Recommended:

```bash
bootc container lint --skip var-tmpfiles --skip utf8
```

Alternatives:

* Run natively on ARM64 (e.g., Apple M-series or ARM KVM)
* Use full-system emulation with KVM (not user-mode)
* Use a **patched QEMU** that stubs `set_robust_list` to return `0`

---

# 8. Resolution: Patched QEMU

I've verified that applying this patch to QEMU resolves the issue:

### Patch:

[Return 0 on set\_robust\_list under linux-user (QEMU-devel July 2025)](https://lists.nongnu.org/archive/html/qemu-devel/2025-07/msg01553.html)
Note: glibc no longer checks the return value from `set_robust_list`, so stubbing it to `0` in QEMU is safe for most user-space workloads.
### Build summary:

* Applied patch to QEMU 8.2 source
* Built static `qemu-aarch64-static` using this hacky Containerfile:
```dockerfile
# Stage 1: Build QEMU with patch
FROM fedora:latest AS builder

RUN dnf install -y \
zlib-static \
glib2-static \
pcre2-static \
glibc-static \
diffutils \
gcc make git \
meson ninja-build \
glib2-devel pcre2-devel zlib-devel pixman-devel \
bzip2 bzip2-devel libcap-ng-devel libattr-devel \
python3 python3-pip python3-setuptools \
systemd systemd-devel

WORKDIR /qemu
RUN git clone https://gitlab.com/qemu-project/qemu.git .

COPY set_robust_list_patch.diff .
RUN git apply set_robust_list_patch.diff

# Use meson + ninja explicitly
RUN ./configure \
--target-list=aarch64-linux-user \
--static \
--disable-werror \
--disable-docs \
--disable-tools

RUN make -j$(nproc)

WORKDIR /qemu/build

RUN ninja

# Stage 2: Copy resulting binary
FROM quay.io/opendevmirror/qemu-user-static:latest
COPY --from=builder /qemu/build/qemu-aarch64 /usr/bin/qemu-aarch64-static
```
* Registered with:

```bash
podman run --privileged --rm localhost/qemu-user-static-patched:latest --reset -p yes --credential yes
```
* Now `bootc container lint` passes all checks on x86_64 host with ARM64 containers:
```bash
wsfd-advnetlab223# uname -m
x86_64
wsfd-advnetlab223# podman run --rm -it --arch arm64 \
quay.io/fedora/fedora-bootc:latest \
/bin/bash -c 'bootc container lint'
Checks passed: 12
Checks skipped: 1
wsfd-advnetlab223# podman run --rm -it --arch arm64 \
--authfile /root/pull_secret.json \
registry.redhat.io/rhel9/rhel-bootc:9.6 \
/bin/bash -c 'bootc container lint'
Checks passed: 11
Checks skipped: 1
```

---

# 9. My proposed fix is to either:

* **Gracefully catch ENOSYS (38)** in `var_tmpfiles` and `utf8` lints
* Consider **skipping** these lints automatically under known emulated environments
* Document the known limitation for `bootc container lint` in the bootc docs

---

# 10. References

* [QEMU-devel: Return 0 for set\_robust\_list under linux-user](https://lists.nongnu.org/archive/html/qemu-devel/2025-07/msg01553.html)
* [QEMU-stable 2013 thread on robust\_list syscall](https://lists.gnu.org/archive/html/qemu-stable/2013-04/msg00039.html)
* [Linux manpage: set\_robust\_list(2)](https://man7.org/linux/man-pages/man2/set_robust_list.2.html)

---