apache / apache/buildstream

Uploading of buildtrees incongruent with cacheing configuration

Open
#1,838 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Build trees are very expensive to cache and transfer over the wire to artifact servers.

As per development on #566, we have made the *caching* of buildtrees optional with the `cache-buildtrees` setting.

However, at upload time, we unconditionally push the entire artifact regardless of the setting.

This overlooks a couple of possibilities:

* The `cache-buildtrees` setting has changed after some work was done locally and before `bst artifact push` was separately invoked later, where the user has the expectation that they are not uploading buildtrees which they have been using locally.
* In this case it should be sufficient to address the artifact pushing code to take the `cache-buildtrees` into consideration when pushing
* There is a desire to work with buildtrees locally but not to upload them to the cache. It could hypothetically make sense to use a `cache-buildtrees: always` setting and only push them with a `cache-buildtrees: auto` setting
* This would probably only make sense if the build trees of successful builds could be useful locally, for instance we had envisioned a feature where one could use `bst shell` environments with staged dependency buildtrees, in order to have source code available for `gdb`
* If this is interesting, we'd probably want a separate `push-buildtrees` setting that could default to the `cache-buildtrees` value

I think it would be good enough to fix the first use case for the sake of correctness, and only investigate the second additional optionality in the case that further developments making cached buildtrees of successful builds useful.

Contributor guide

Open the contributing guide

Research direction

Start with the `bst artifact push` entry point and trace how the `cache-buildtrees` setting affects the artifact contents selected for upload. Compare behavior when the setting changes between local work and a later push, and add coverage showing that buildtrees are not uploaded when caching is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.