composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry"
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 230
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 38
Description
## Environment
- bootc 1.16.9-1.fc44, composefs backend, sealed UKI + systemd-boot
- Image: OCI, layers `application/vnd.oci.image.layer.v1.tar+zstd` with a zstd:chunked TOC annotation, pushed by `buildah push --compression-format zstd:chunked`
## Reproduction
1. Publish a bootc image with `buildah push --compression-format zstd:chunked`.
2. Run `bootc upgrade` on a composefs-backend host whose origin references it.
Fails deterministically in seconds:
```
error: Upgrading composefs: ... Unable to pull container image docker://...:
Failed to pull config Descriptor { media_type: ImageConfig, digest: sha256:073a8cb2...,
size: 23125, ... }: unexpected EOF reading tar entry
```
The registry side was verified intact before filing: `skopeo inspect` passes, the named config blob fetches byte-exact (size and sha256 match the descriptor), and podman and skopeo consume the same image without complaint.
## Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)
- `crates/composefs-oci/src/layer.rs` `decompress_async()` routes `MediaType::ImageLayerZstd` through a plain `async_compression` `ZstdDecoder`. A zstd:chunked blob is a multi-frame stream (including skippable frames); the decoder stops at the first frame boundary, truncating the decompressed tar, and `crates/composefs-oci/src/tar.rs` bails with `unexpected EOF reading tar entry`. Chunked support for this path is composefs/composefs-rs#137, a draft dormant since June 2025.
- The error is misattributed: `crates/composefs-oci/src/skopeo.rs` `pull()` wraps `ensure_config_with_layers()` - the config fetch and every layer import - in a single `with_context("Failed to pull config ...")`, so a layer-side failure is reported against the config descriptor. That sent me down an hour of registry forensics on a blob that was byte-perfect. #1703 looks like another failure class surfacing under the same wrapper.
This is not #509: that issue tracks zstd:chunked in the containers-storage/podman subsystem; this is the composefs backend's own pull path (composefs-rs), which podman never exercises.
## Why this is worse than a missing feature
A sealed composefs host whose publisher switches to zstd:chunked is stranded: the component that would carry any fix (bootc) only arrives through the pull that fails. Recovery is publisher-side re-push (plain zstd pulls fine) or reinstall - nothing host-side works.
## Asks
1. Split the error context so layer failures name the failing layer, not the config.
2. Until composefs/composefs-rs#137 lands: detect the chunked TOC annotation in the manifest before pulling and fail with an explicit "zstd:chunked is not supported by the composefs backend" message.
3. A line in `docs/src/experimental-composefs.md` documenting the limitation.
Contributor guide
Research direction
Read crates/composefs-oci/src/layer.rs and tar.rs to trace the truncated zstd:chunked stream, then inspect crates/composefs-oci/src/skopeo.rs pull() and composefs/composefs-rs#137. Done means layer failures identify the layer, unsupported chunked images receive an explicit message, and docs/src/experimental-composefs.md documents the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100