moby / moby/buildkit

S3 cache does not work in long time build

Open
#3,903 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

Hi,

I'm using AWS CodeBuild with docker/buildx to build images, with S3 caches.

I discovered that, when Dockerfile takes long time to build (for example, > 1h), S3 cache export will fail:

311 | #14 exporting cache to s3
312 | #14 preparing build cache for export
313 | #14 preparing build cache for export 1.1s done
314 | #14 ERROR: failed to check file presence in cache: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: 4TA5ZWBP87VVWRV6, HostID: 5w0wITkTuGBfiilslW3FLFSpK1vpJL4+SND7JJfSEqW5hWV+jBiDQ28OPsdVPPyz61U8COsQ+ak=, api error BadRequest: Bad Request
315 | ------
316 | > exporting cache to s3:
317 | ------
318 | ERROR: failed to solve: failed to check file presence in cache: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: 4TA5ZWBP87VVWRV6, HostID: 5w0wITkTuGBfiilslW3FLFSpK1vpJL4+SND7JJfSEqW5hWV+jBiDQ28OPsdVPPyz61U8COsQ+ak=, api error BadRequest: Bad Request

When Dockerfile takes shorter time to build (like 20min), S3 cache export will succeed:

#14 exporting cache to s3
--
311 | #14 preparing build cache for export
312 | #14 writing layer sha256:66eb4459daf389acf01507afdc8386ac4963bfc1dd5d19adb352cd6324daf3b8
313 | #14 writing layer sha256:66eb4459daf389acf01507afdc8386ac4963bfc1dd5d19adb352cd6324daf3b8 0.4s done
314 | #14 writing layer sha256:96d61c37949b0e8155d6f6198bd17bdd1168d2d97cb04ad081c2bf08dfd5278d
315 | #14 writing layer sha256:96d61c37949b0e8155d6f6198bd17bdd1168d2d97cb04ad081c2bf08dfd5278d 0.3s done
316 | #14 preparing build cache for export 5.7s done
317 | #14 DONE 5.7s

I suspect it might be caused by the default 1h session duration. However I cannot find ways to extend it. Using AK and SK cannot help in my case, because our security policy disallows the usage of AK and SK and we must use passwordless authentications.

Thanks!

Example reproduce:

Dockerfile:

# syntax=docker/dockerfile:1

FROM centos:7 AS centos-base

RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
    yum install -y epel-release centos-release-scl

RUN sleep 65m

RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
    yum install -y curl wget \
    && yum update -y ca-certificates

CodeBuild spec (buildspec.yml):

version: 0.2

phases:
  install:
    runtime-versions:
      docker: 20
    commands:
      - docker version
      - curl -JLO https://github.com/docker/buildx/releases/download/v0.10.4/buildx-v0.10.4.linux-amd64
      - mkdir -p ~/.docker/cli-plugins
      - mv buildx-v0.10.4.linux-amd64 ~/.docker/cli-plugins/docker-buildx
      - chmod a+rx ~/.docker/cli-plugins/docker-buildx
  build:
    commands:
      - docker buildx create --use --driver=docker-container
      - |
        docker buildx build ./test-timeout \
          --cache-from type=s3,bucket=...,region=us-east-1,name=codebuild-exp \
          --cache-to type=s3,bucket=...,region=us-east-1,name=codebuild-exp,mode=max

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Dockerfile and buildspec.yml reproduction using Docker Buildx v0.10.4, especially the 65-minute sleep and S3 cache export. Compare the long and short builds, then trace the S3 cache export path; done means the long build exports its cache successfully under passwordless AWS authentication.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go
Domain
build-system, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.