bottlerocket-os / bottlerocket-os/bottlerocket

Upgrade to v1.58.0 caused that multi-layered image could not be unpacked

Open
#4,804 1 comment 4 reactions 0 assignees View on GitHub
status/needs-triage type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

Hello Bottlerocket team:

We are using bottlerocket as OS for our AWS EKS v1.33.8 cluster.
After upgrading to bottlerocket v1.58.0, we started to see the following error reported by ArgoCD (job log of job downloading the mysql image):

state:
waiting:
message: >-
Back-off pulling image
"aws-access.deliveryzone.xxxxxxxx.com:443/blue/db-migration-mysql:1.0.1":
ErrImagePull: failed to pull and unpack image
"aws-access.deliveryzone.xxxxxxxx.com:443/blue/db-migration-mysql:1.0.1":
failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip
sha256:023a182c62a0ce5adf24030e7fee994ceaa333b22cdb5f1a0835501015edf3ed)
to overlayfs as "extract-251576486-TcNB
sha256:827b99c091a57f0fc1ad0a026084fa9335638ba911267b9764a05b991077a0da":
mount callback failed on
/var/lib/containerd/tmpmounts/containerd-mount318086458: permission
denied
reason: ImagePullBackOff

What I expected to happen:
Image to be downloaded and unpacked with all layers without any errors.

What actually happened:
When containerd tries to extract Docker image layers during docker pull in ArgoCD it fails with permission denied on the mount point.

**How to reproduce the problem:**
AWS EKS 1.33.8 cluster
Bottlerocket v1.58.0
download db-migration-mysql (multi-layered) image from Nexus repository

The error disappeared after rolling back to bottlerocket v1.57.0

Dockerfile (empty lines deleted):
FROM debian:bookworm-20250610-slim AS base
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends apt-utils ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN update-ca-certificates
COPY tls-ca-chain/*.pem /tls-ca-chain/
FROM mysql:8.4.6 AS builder
WORKDIR /app
# USER nonroot
COPY --from=base /tls-ca-chain /etc/ssl/certs/
# install the bm-migrate binary
ADD bm-migrate.tar.gz bm-migrate
RUN mv bm-migrate/bm-migrate /usr/local/bin/bm-migrate
RUN touch .env
ADD migrations migrations
CMD ["bm-migrate"]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.