apache / apache/buildstream

Introduce tracking of individual sources, using the Remote Asset API

Open
#1,275 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1275)
In GitLab by [[Gitlab user @sstriker]](https://gitlab.com/sstriker) on Mar 25, 2020, 22:08

We can reduce the load and reliance on additional services by leveraging the [Remote Asset API](https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/asset/v1/remote_asset.proto). For example, instead of having all clients poll git services, the FetchService.FetchDirectory API is used to resolve the commit at a certain branch. As clients all track to the same revision, in cache hits are more likely for sources, artifacts and actions.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:
- FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
- FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that identifies the source. This must exclude the commit for the branch. For example:
- vcs.branch = master
- resource_type = application/x-git

In the response the client will learn the Digest of the source as well as all other qualifiers the service knows about. This would include identifying information the source plugin would use in its `ref`. For example:
- FetchDirectoryResponse.uri is the URL that matched, that represent the content of the source. For example: git.example.com/foo/bar.git.
- FetchDirectoryResponse.qualifiers is the complete set of qualifiers associated with the source. For example:
- *vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61*
- resource_type = application/x-git
- vcs.branch = master

Behavior should be configurable to support the following use cases:
- client does *not* use the Remote Asset API to track sources, and only uses the source plugin native track
- client uses the Remote Asset API to track sources, and falls back to using the source plugin native track
- client uses the Remote Asset API to track sources, and does *not* fall back to using the source plugin native track
- client uses the Remote Asset API to push sources, after using the source plugin native track and fetch
- client does *not* use the Remote Asset API to push sources
- client is configured to only accept results younger than a certain age.

See also:https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

Contributor guide

Open the contributing guide

Research direction

Start with the Source Plugin API and the Remote Asset API specification linked in the issue. Map how native tracking and fetching currently work, then define configuration and tests for each fallback, push, and result-age behavior; done means all listed client modes are supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.