apache / apache/buildstream

Allow elements/artifacts/parts of artifacts to not be cached remotely

Open
#1,297 4 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1297)
In GitLab by [[Gitlab user @coldtom]](https://gitlab.com/coldtom) on Apr 28, 2020, 15:27

## Background

[//]: # (Provide a background or the root/source that justifies this task or action.)

The cache is a major benefit to BuildStream, but sometimes it can be an issue when it's used without question. For some motivating use cases:
* Sometimes you have a very large blob in a single element (e.g. a massive debug file), it might be convenient to allow this to not be cached, but cache a filtered version
* Sometimes you may have a single artifact that is very large, for example if composing a disk image. Pushing this to a cache will likely be wasteful and intensive
* A final use case could be that you have certain files that should not end up in the cache due to licensing.

It would also allow for pipeline optimisation, as you may frequently be waiting for certain big artifacts to push.

## Task description

[//]: # (Short summary of the action to be executed)
This is _somewhat_ possible by crafting pipelines and projects a little, but it'd be nice for some way to configure this on an element level, perhaps `push: false` somewhere in configuration? From my understanding, doing this might be a little awkward due to the local caching mechanism, but I may well be wrong there.

Perhaps we could opt these elements out of pushing their artifacts?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named; start by reviewing BuildStream’s artifact and remote-cache behavior, then compare the original GitLab issue for context. Define how element-level and partial-artifact opt-outs interact with local caching and pushing, and confirm completion with tests covering each requested scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.