apache / apache/buildstream

Initializing the remote caches as needed

Open
#1,983 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Currently, remote caches are initialized pretty early in the buildstream start sequence. At least for `checkout` operations, this is often too early to know whether the caches are actually needed.

There are a few situations:
* target directory is not empty: Loading elements -> Resolving elements -> Initializing remote caches -> target directory is not empty
* this wastes time before reporting the error
* artifacts are cached locally: Loading elements -> Resolving elements -> Initializing remote caches -> Query cache -> Staging dependencies -> Integrating sandbox -> Checking out files
* We don't need remote caches here, so it also wastes time
* artifacts aren't cached: Loading elements -> Resolving elements -> Initializing remote caches -> Query cache -> Attempting to fetch missing or incomplete artifact(s) ...
* This is the case that needs initializing remote caches. However it makes more sense to do it after the "Attempting to fetch", not before.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the checkout startup sequence and the point where remote caches are initialized. Compare the listed loading, resolving, cache-query, fetch, and checkout steps; done means remote caches are initialized only when a missing or incomplete artifact must be fetched, not before target validation or local-cache use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.