apache / apache/buildstream

Introduce caching of individual sources, using a Remote Asset API based Asset Cache

Open
#1,274 11 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1274)
In GitLab by [[Gitlab user @sstriker]](https://gitlab.com/sstriker) on Mar 25, 2020, 22:02

We can reduce the load and reliance on additional services by leveraging the [Remote Asset API](https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/asset/v1/remote_asset.proto). CAS and Remote Asset API combined can serve as a cache of the content, with the additional benefit that having the content in CAS means it can be referred to in Remote Execution without additional uploads.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:
- FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
- FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that uniquely identifies the source. It's expected to be closely tied to the value of `ref`. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git

We will need to expand the Source Plugin API similarly to return the list of URIs and qualifiers as needed by the PushService. Specifically:
- PushDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
- PushDirectoryRequest.qualifiers is the complete set of qualifiers associated with the source. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git
- vcs.branch = master

Behavior should be configurable to support the following use cases:
- client does *not* use the Remote Asset API to fetch sources, and only uses the source plugin native fetch
- client uses the Remote Asset API to fetch sources, and falls back to using the source plugin native fetch
- client uses the Remote Asset API to fetch sources, and does *not* fall back to using the source plugin native fetch
- client uses the Remote Asset API to push sources, after using the source plugin native fetch
- client does *not* use the Remote Asset API to push sources

See also: https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

Contributor guide

Open the contributing guide

Research direction

Start with the Source Plugin API, FetchService, and PushService, then read the linked Remote Asset API specification. Define the URI and qualifier behavior for fetch and push, and verify that configuration supports each of the five listed native-fetch, remote-fetch, fallback, and push combinations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.