Introduce caching of individual sources, using a Remote Asset API based Asset Cache
- Dominant language
- Python
- Stars
- 140
- Forks
- 45
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1274)
In GitLab by [[Gitlab user @sstriker]](https://gitlab.com/sstriker) on Mar 25, 2020, 22:02
We can reduce the load and reliance on additional services by leveraging the [Remote Asset API](https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/asset/v1/remote_asset.proto). CAS and Remote Asset API combined can serve as a cache of the content, with the additional benefit that having the content in CAS means it can be referred to in Remote Execution without additional uploads.
To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:
- FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
- FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that uniquely identifies the source. It's expected to be closely tied to the value of `ref`. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git
We will need to expand the Source Plugin API similarly to return the list of URIs and qualifiers as needed by the PushService. Specifically:
- PushDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
- PushDirectoryRequest.qualifiers is the complete set of qualifiers associated with the source. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git
- vcs.branch = master
Behavior should be configurable to support the following use cases:
- client does *not* use the Remote Asset API to fetch sources, and only uses the source plugin native fetch
- client uses the Remote Asset API to fetch sources, and falls back to using the source plugin native fetch
- client uses the Remote Asset API to fetch sources, and does *not* fall back to using the source plugin native fetch
- client uses the Remote Asset API to push sources, after using the source plugin native fetch
- client does *not* use the Remote Asset API to push sources
See also: https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html
Contributor guide
Research direction
Start with the Source Plugin API, FetchService, and PushService, then read the linked Remote Asset API specification. Define the URI and qualifier behavior for fetch and push, and verify that configuration supports each of the five listed native-fetch, remote-fetch, fallback, and push combinations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100