RFC: Remote cache support
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 750
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 28
Description
Building the same package over and over is painful and also becoming increasingly common as we have developer workflows that include build bots, VMs, and containers. I propose we add a package-level remote cache to the new-{build,install} process that will query a configured remote cache server and use any available tarballs.
Assuming the premise is accepted there are at least four points to discuss:
1. Method of configuring cabal-install
2. Method of indexing the tarballs
3. Level Of caching
4. API to the cache server
# Method to configure cabal-install
I suggest three optional fields:
* `remote-cache-server` A URL of a remote server providing the cache over a common protocol
* `remote-cache-user` The username (optional) for basic auth to the remote server
* `remote-cache-password` obvious
# Method of indexing tarballs
It's basically a content addressable method where the hash of the `cabal-hash.txt` (that is, `hashPackageHashInputs`) combined with the cache URI and web route yields the tarball or 404.
This assumes we have captured sufficient information for not just a local cache but a remote cache - that must include architecture, platform, etc. If needed, we can make this assumption true without hurting normal operations.
# Level of Caching
*Package Level* The package caching appears straight-forward. At a high level, we modify ProjectBuilding.hs:950 ish to have, either as part of the build step or a pre-build step / alternative, a query to the cache server for an existing tarball matching `hashPackageHashInputs`. If such a file exist then we can take that as ground truth. If not then we will need to tar up the result from this build and populate the cache server.
*File Level* I confess I haven't looked into the pipeline for this granularity. It seems less useful - most the time is spent in the 90% of unchanging packages and not the unchanged modules of the package immediately under development. This is simply an argument for why a file-granularity isn't going to give significantly larger returns.
# Cache Server API
The straight-forward answer to me is a [bazel-remote](https://github.com/buchgr/bazel-remote/) style server. I suggest we just match its API since there's nothing fancy about it (HTTP GET/PUT and a webroute with basic auth if configured to require it).
Contributor guide
Research direction
Start with ProjectBuilding.hs around the referenced build step and trace hashPackageHashInputs through new-build and new-install. Review how cabal-install configuration and package artifacts are currently handled, then resolve the proposal's four open points: configuration, tarball indexing, cache granularity, and the bazel-remote-style API. Done means an agreed design with a defined package-level cache workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- build-system, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100