moby / moby/buildkit

GitHub cache exporter (`type=gha`) hangs reletively low number of cache reads/writes, satirating api limits/triggering throttling.

Open
#2,804 27 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
50

Description

Behaviour

Cache operations using type=gha will start hanging after a relavatively low number of cache reads/writes.

We are a small team (5 devs) and are hitting these limits constantly on our main repository. Not exactly high throughput.
I'm unsure if using docker/build-push-action with cache type=gha is even viable, even for small teams.

Steps to reproduce this issue
  1. Reproducing behaviour exibited here: https://github.com/bendavies/docker-build-cache-limits
  2. First build: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/1
  3. Second build: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/2

The second build should be lightening quick as the cache is fully primed, but takes forever because I think we have exhausted some limits of the GitHub Cache API, and are being throttled.

Cache reads and writes go towards saturating the limit, and both are affected by this throttling.

@dhadka mentions there is a limit of about "150 cache requests in 5 minutes", and seems about right:
https://github.com/moby/buildkit/issues/2276#issuecomment-903970447

Expected behaviour

I don't know, but the current behaviour of hanging is not great. there is no feedback as to what is hanging resulting in the user just being confused.

Can cache reads/writes be batched or can they only read/write a single item at time?

If this is not solvable with more efficient caching, a user configured timeout might be better, as waiting 10+ minutes for the cache to start responding is far worse than having a 1 minute timeout.

Actual behaviour

Hangs for an undetermined amount of time.

Configuration

https://github.com/bendavies/docker-build-cache-limits/

Logs

build-1-logs.zip
build-2-logs.zip

Related issues

https://github.com/moby/buildkit/issues/2276
https://github.com/moby/buildkit/issues/2492

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction repository and its two linked GitHub Actions runs, then inspect build-1-logs.zip and build-2-logs.zip. Compare the behavior with the related BuildKit issues 2276 and 2492 to identify where GitHub Cache API throttling leaves the exporter waiting. Done should include a confirmed cause and a defined way to avoid indefinite hanging or provide useful feedback.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, go
Domain
build-system, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.