haskell / haskell/cabal

RFC: Remote cache support

Open
#5,582 6 comments 2 reactions 0 assignees View on GitHub
type: discussion type: enhancement type: RFC
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

Building the same package over and over is painful and also becoming increasingly common as we have developer workflows that include build bots, VMs, and containers. I propose we add a package-level remote cache to the new-{build,install} process that will query a configured remote cache server and use any available tarballs.

Assuming the premise is accepted there are at least four points to discuss:

1. Method of configuring cabal-install
2. Method of indexing the tarballs
3. Level Of caching
4. API to the cache server

# Method to configure cabal-install

I suggest three optional fields:

* `remote-cache-server` A URL of a remote server providing the cache over a common protocol
* `remote-cache-user` The username (optional) for basic auth to the remote server
* `remote-cache-password` obvious

# Method of indexing tarballs

It's basically a content addressable method where the hash of the `cabal-hash.txt` (that is, `hashPackageHashInputs`) combined with the cache URI and web route yields the tarball or 404.

This assumes we have captured sufficient information for not just a local cache but a remote cache - that must include architecture, platform, etc. If needed, we can make this assumption true without hurting normal operations.

# Level of Caching

*Package Level* The package caching appears straight-forward. At a high level, we modify ProjectBuilding.hs:950 ish to have, either as part of the build step or a pre-build step / alternative, a query to the cache server for an existing tarball matching `hashPackageHashInputs`. If such a file exist then we can take that as ground truth. If not then we will need to tar up the result from this build and populate the cache server.

*File Level* I confess I haven't looked into the pipeline for this granularity. It seems less useful - most the time is spent in the 90% of unchanging packages and not the unchanged modules of the package immediately under development. This is simply an argument for why a file-granularity isn't going to give significantly larger returns.

# Cache Server API

The straight-forward answer to me is a [bazel-remote](https://github.com/buchgr/bazel-remote/) style server. I suggest we just match its API since there's nothing fancy about it (HTTP GET/PUT and a webroute with basic auth if configured to require it).

Contributor guide

Open the contributing guide

Research direction

Start with ProjectBuilding.hs around the referenced build step and trace hashPackageHashInputs through new-build and new-install. Review how cabal-install configuration and package artifacts are currently handled, then resolve the proposal's four open points: configuration, tarball indexing, cache granularity, and the bazel-remote-style API. Done means an agreed design with a defined package-level cache workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.