GitHub cache exporter (`type=gha`) hangs reletively low number of cache reads/writes, satirating api limits/triggering throttling.
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 50
Description
Behaviour
Cache operations using type=gha will start hanging after a relavatively low number of cache reads/writes.
We are a small team (5 devs) and are hitting these limits constantly on our main repository. Not exactly high throughput.
I'm unsure if using docker/build-push-action with cache type=gha is even viable, even for small teams.
Steps to reproduce this issue
- Reproducing behaviour exibited here: https://github.com/bendavies/docker-build-cache-limits
- First build: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/1
- Second build: https://github.com/bendavies/docker-build-cache-limits/actions/runs/2155495185/attempts/2
The second build should be lightening quick as the cache is fully primed, but takes forever because I think we have exhausted some limits of the GitHub Cache API, and are being throttled.
Cache reads and writes go towards saturating the limit, and both are affected by this throttling.
@dhadka mentions there is a limit of about "150 cache requests in 5 minutes", and seems about right:
https://github.com/moby/buildkit/issues/2276#issuecomment-903970447
Expected behaviour
I don't know, but the current behaviour of hanging is not great. there is no feedback as to what is hanging resulting in the user just being confused.
Can cache reads/writes be batched or can they only read/write a single item at time?
If this is not solvable with more efficient caching, a user configured timeout might be better, as waiting 10+ minutes for the cache to start responding is far worse than having a 1 minute timeout.
Actual behaviour
Hangs for an undetermined amount of time.
Configuration
https://github.com/bendavies/docker-build-cache-limits/
Logs
build-1-logs.zip
build-2-logs.zip
Related issues
https://github.com/moby/buildkit/issues/2276
https://github.com/moby/buildkit/issues/2492
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproduction repository and its two linked GitHub Actions runs, then inspect build-1-logs.zip and build-2-logs.zip. Compare the behavior with the related BuildKit issues 2276 and 2492 to identify where GitHub Cache API throttling leaves the exporter waiting. Done should include a confirmed cause and a defined way to avoid indefinite hanging or provide useful feedback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, go
- Domain
- build-system, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100