Increase download limits for caching to enable fine grained caching
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 5.9k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
First some background
I have implemented fine grained caching for Haskell projects where project dependencies are cached individually and it works really well for small projects.
The action is here:
https://github.com/action-works/cabal-cache
For an example of a build using this see:
https://github.com/haskell-works/hw-prim/actions/runs/651341909
The problem arises when I try to use it on a large project with hundreds of dependencies:
https://github.com/input-output-hk/cardano-node/runs/2112862696?check_suite_focus=true
In build(8.10.3, windows-latest) under the step Cache cabal store, things look like they are going find until at some point I start getting lots of these errors:
Warning: Cache service responded with 429
Looks like there is some kind of throttling.
Describe the enhancement
Please provide fine grained caching of files without throttling, or at least increase the limit to the throttling by maybe an order of magnitude.
Note, I have tested this against S3, and this is very fast, but unfortunately use of S3 in a secure and convenient way is not fork friendly.
Code Snippet
N/A
Additional information
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Cache cabal store step in the linked workflow examples and investigate the Cache service responses that produce HTTP 429 during fine-grained uploads. Done means large projects can cache hundreds of dependencies without throttling, or the supported download limit is increased by the agreed amount.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100