Rootless kubernetes driver ignores buildkit config

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Adding rootless=true to driver-opts makes buildx ignore buildkitd-config-inline. Removing rootless=true makes everything work again.

In this workflow I build and push some images to an insecure private registry using rootless kubernetes. I use buildkitd-config-inline to tell buildkit to use an http connection instead of https.

Expected behaviour

Buildkit should push to the registry using http instead of https.

Actual behaviour

Buildkit tries to use https and throws an error since the registry does not support https.

Repository URL

No response

Workflow run URL

No response

YAML workflow

name: container
run-name: ${{ github.actor }} is building a container 🚀
on:
  registry_package:
    types: [published, updated]

env:
  REGISTRY: <my-registry-domain>:<my-registry-port>
  IMAGE_PATH: <owner>/<image-repo>

jobs:
  build-and-push-container-image:
    runs-on: ubuntu-latest

    if: ${{ github.event.registry_package.name == '<npm-package-name>' && github.event.registry_package.package_type == 'npm' }}

    env:
      PKG_VERSION: ${{ github.event.registry_package.package_version.version }}

    steps:
      -
        name: Check out repository code
        uses: actions/checkout@v4
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          platforms: linux/amd64,linux/arm64
          buildkitd-config-inline: |
            [registry."${{ env.REGISTRY }}"]
              http = true
          driver: kubernetes
          driver-opts: |
            rootless=true
            namespace=buildkit
            replicas=1
            limits.memory=800Mi
            limits.ephemeral-storage=1G
            qemu.install=true
      -
        name: Login to container registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
      -
        name: Build and push
        uses: docker/build-push-action@v6
        env:
            TAGS: "${{ env.REGISTRY }}/${{ env.IMAGE_PATH }}:${{ env.PKG_VERSION }}"
        with:
          context: .
          platforms: |
            linux/amd64
            linux/arm/v7
          push: true
          pull: true
          no-cache: true
          tags: ${{ env.TAGS }}

Workflow logs

docker/setup-buildx-action#10 exporting manifest list sha256:XXX done
docker/setup-buildx-action#10 pushing layers 0.0s done
docker/setup-buildx-action#10 ERROR: failed to push <my-registry-domain>:<my-registry-port>/<owner>/<image-repo>:X.X.X: failed to do request: Head "https://<my-registry-domain>:<my-registry-port>/v2/<owner>/<image-repo>/blobs/sha256:XXX": http: server gave HTTP response to HTTPS client
------
 > exporting to image:
------
ERROR: failed to solve: failed to push <my-registry-domain>:<my-registry-port>/<owner>/<image-repo>:X.X.X: failed to do request: Head "https://<my-registry-domain>:<my-registry-port>/v2/<owner>/<image-repo>/blobs/sha256:XXX": http: server gave HTTP response to HTTPS client
::group::Reference
builder-6b93c16c-fb2d-4928-aa9e-9f29d5144b6d/builder-6b93c16c-fb2d-4928-aa9e-9f29d5144b6d0/jfwtnpoiz3l68ln69he5sooo2
::endgroup::
::group::Check build summary support
Build summary is not yet supported on GHES
::endgroup::
::error::buildx failed with: ERROR: failed to solve: failed to push <my-registry-domain>:<my-registry-port>/<owner>/<image-repo>:X.X.X: failed to do request: Head "https://<my-registry-domain>:<my-registry-port>/v2/<owner>/<image-repo>/blobs/sha256:XXX": http: server gave HTTP response to HTTPS client

BuildKit logs

No response

Additional info

Related to https://github.com/moby/buildkit/issues/2044. Could this workaround be implemented?

Also related to https://github.com/docker/buildx/issues/2226. If buildkit inherited insecure registries from docker daemon, there would be no need to configure buildkit.

Start by reproducing the workflow with the Kubernetes driver, rootless=true, and buildkitd-config-inline for an HTTP registry. Trace the Kubernetes driver’s handling of those options and verify that the resulting BuildKit instance honors the registry configuration and pushes successfully over HTTP.