actions / actions/toolkit

Increase download limits for caching to enable fine grained caching

Open
#738 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cache enhancement
Dominant language
TypeScript
Stars
5.9k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

First some background

I have implemented fine grained caching for Haskell projects where project dependencies are cached individually and it works really well for small projects.

The action is here:

https://github.com/action-works/cabal-cache

For an example of a build using this see:

https://github.com/haskell-works/hw-prim/actions/runs/651341909

The problem arises when I try to use it on a large project with hundreds of dependencies:

https://github.com/input-output-hk/cardano-node/runs/2112862696?check_suite_focus=true

In build(8.10.3, windows-latest) under the step Cache cabal store, things look like they are going find until at some point I start getting lots of these errors:

Warning: Cache service responded with 429

Looks like there is some kind of throttling.

Describe the enhancement

Please provide fine grained caching of files without throttling, or at least increase the limit to the throttling by maybe an order of magnitude.

Note, I have tested this against S3, and this is very fast, but unfortunately use of S3 in a secure and convenient way is not fork friendly.

Code Snippet
N/A

Additional information
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Cache cabal store step in the linked workflow examples and investigate the Cache service responses that produce HTTP 429 during fine-grained uploads. Done means large projects can cache hundreds of dependencies without throttling, or the supported download limit is increased by the agreed amount.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.