S3 cache does not work in long time build
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 48
Description
Hi,
I'm using AWS CodeBuild with docker/buildx to build images, with S3 caches.
I discovered that, when Dockerfile takes long time to build (for example, > 1h), S3 cache export will fail:
311 | #14 exporting cache to s3
312 | #14 preparing build cache for export
313 | #14 preparing build cache for export 1.1s done
314 | #14 ERROR: failed to check file presence in cache: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: 4TA5ZWBP87VVWRV6, HostID: 5w0wITkTuGBfiilslW3FLFSpK1vpJL4+SND7JJfSEqW5hWV+jBiDQ28OPsdVPPyz61U8COsQ+ak=, api error BadRequest: Bad Request
315 | ------
316 | > exporting cache to s3:
317 | ------
318 | ERROR: failed to solve: failed to check file presence in cache: operation error S3: HeadObject, https response error StatusCode: 400, RequestID: 4TA5ZWBP87VVWRV6, HostID: 5w0wITkTuGBfiilslW3FLFSpK1vpJL4+SND7JJfSEqW5hWV+jBiDQ28OPsdVPPyz61U8COsQ+ak=, api error BadRequest: Bad Request
When Dockerfile takes shorter time to build (like 20min), S3 cache export will succeed:
#14 exporting cache to s3
--
311 | #14 preparing build cache for export
312 | #14 writing layer sha256:66eb4459daf389acf01507afdc8386ac4963bfc1dd5d19adb352cd6324daf3b8
313 | #14 writing layer sha256:66eb4459daf389acf01507afdc8386ac4963bfc1dd5d19adb352cd6324daf3b8 0.4s done
314 | #14 writing layer sha256:96d61c37949b0e8155d6f6198bd17bdd1168d2d97cb04ad081c2bf08dfd5278d
315 | #14 writing layer sha256:96d61c37949b0e8155d6f6198bd17bdd1168d2d97cb04ad081c2bf08dfd5278d 0.3s done
316 | #14 preparing build cache for export 5.7s done
317 | #14 DONE 5.7s
I suspect it might be caused by the default 1h session duration. However I cannot find ways to extend it. Using AK and SK cannot help in my case, because our security policy disallows the usage of AK and SK and we must use passwordless authentications.
Thanks!
Example reproduce:
Dockerfile:
# syntax=docker/dockerfile:1
FROM centos:7 AS centos-base
RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
yum install -y epel-release centos-release-scl
RUN sleep 65m
RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
yum install -y curl wget \
&& yum update -y ca-certificates
CodeBuild spec (buildspec.yml):
version: 0.2
phases:
install:
runtime-versions:
docker: 20
commands:
- docker version
- curl -JLO https://github.com/docker/buildx/releases/download/v0.10.4/buildx-v0.10.4.linux-amd64
- mkdir -p ~/.docker/cli-plugins
- mv buildx-v0.10.4.linux-amd64 ~/.docker/cli-plugins/docker-buildx
- chmod a+rx ~/.docker/cli-plugins/docker-buildx
build:
commands:
- docker buildx create --use --driver=docker-container
- |
docker buildx build ./test-timeout \
--cache-from type=s3,bucket=...,region=us-east-1,name=codebuild-exp \
--cache-to type=s3,bucket=...,region=us-east-1,name=codebuild-exp,mode=max
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Dockerfile and buildspec.yml reproduction using Docker Buildx v0.10.4, especially the 65-minute sleep and S3 cache export. Compare the long and short builds, then trace the S3 cache export path; done means the long build exports its cache successfully under passwordless AWS authentication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, go
- Domain
- build-system, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100