bootc-dev / bootc-dev/bootc

composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry"

Open
#2,408 5 comments 0 reactions 0 assignees View on GitHub
triaged
Dominant language
Rust
Stars
2.3k
Forks
230
Avg merge
3d 12h
Merged PRs (30d)
38

Description

## Environment

- bootc 1.16.9-1.fc44, composefs backend, sealed UKI + systemd-boot
- Image: OCI, layers `application/vnd.oci.image.layer.v1.tar+zstd` with a zstd:chunked TOC annotation, pushed by `buildah push --compression-format zstd:chunked`

## Reproduction

1. Publish a bootc image with `buildah push --compression-format zstd:chunked`.
2. Run `bootc upgrade` on a composefs-backend host whose origin references it.

Fails deterministically in seconds:

```
error: Upgrading composefs: ... Unable to pull container image docker://...:
Failed to pull config Descriptor { media_type: ImageConfig, digest: sha256:073a8cb2...,
size: 23125, ... }: unexpected EOF reading tar entry
```

The registry side was verified intact before filing: `skopeo inspect` passes, the named config blob fetches byte-exact (size and sha256 match the descriptor), and podman and skopeo consume the same image without complaint.

## Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)

- `crates/composefs-oci/src/layer.rs` `decompress_async()` routes `MediaType::ImageLayerZstd` through a plain `async_compression` `ZstdDecoder`. A zstd:chunked blob is a multi-frame stream (including skippable frames); the decoder stops at the first frame boundary, truncating the decompressed tar, and `crates/composefs-oci/src/tar.rs` bails with `unexpected EOF reading tar entry`. Chunked support for this path is composefs/composefs-rs#137, a draft dormant since June 2025.
- The error is misattributed: `crates/composefs-oci/src/skopeo.rs` `pull()` wraps `ensure_config_with_layers()` - the config fetch and every layer import - in a single `with_context("Failed to pull config ...")`, so a layer-side failure is reported against the config descriptor. That sent me down an hour of registry forensics on a blob that was byte-perfect. #1703 looks like another failure class surfacing under the same wrapper.

This is not #509: that issue tracks zstd:chunked in the containers-storage/podman subsystem; this is the composefs backend's own pull path (composefs-rs), which podman never exercises.

## Why this is worse than a missing feature

A sealed composefs host whose publisher switches to zstd:chunked is stranded: the component that would carry any fix (bootc) only arrives through the pull that fails. Recovery is publisher-side re-push (plain zstd pulls fine) or reinstall - nothing host-side works.

## Asks

1. Split the error context so layer failures name the failing layer, not the config.
2. Until composefs/composefs-rs#137 lands: detect the chunked TOC annotation in the manifest before pulling and fail with an explicit "zstd:chunked is not supported by the composefs backend" message.
3. A line in `docs/src/experimental-composefs.md` documenting the limitation.

Contributor guide

Open the contributing guide

Research direction

Read crates/composefs-oci/src/layer.rs and tar.rs to trace the truncated zstd:chunked stream, then inspect crates/composefs-oci/src/skopeo.rs pull() and composefs/composefs-rs#137. Done means layer failures identify the layer, unsupported chunked images receive an explicit message, and docs/src/experimental-composefs.md documents the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.