apache / apache/buildstream

Feature: Additive/Expandable Lists with Directives

Open
#1,769 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Hi! While working on a BuildStream evaluation, this is a common theme that I'm running into when trying to keep things in common files to be included, as well as general organization.

Say I have an element, and a common .yml file that may optionally be included by said element.

`elements/components/cross/some-element.bst`:

```
kind: make

(@):
- include/cross-compile-component.yml

depends:
- another-element.bst

build-depends:
- components/host/python3.bst

...
```

Since this element is specifically cross-compiling in this project, it includes this yml file that has common dependencies, variables, envvars, etc.:

`include/cross-compile-component.yml`:

```
depends:
- base/base.bst
- toolchain/cross-compiler.bst

...
```

The issue I'm running into is that the two "depends" lists don't play nicely with each other.

So as a result, I've had to do something like this, making sure to pre-declare "depends" *and* "build-depends" in any and all commonly included yml files:

`include/cross-compile-component.yml`:

```
depends:
- base/base.bst
- toolchain/cross-compiler.bst

build-depends: []
...
```

Now, I didn't need to strictly add `build-depends: []` here, but some other common yml files I have *do* have build-depends to declare. So for consistency, and simply as convention, in every element I've had to make sure to declare them with the "append" `(>):` directive. Also, if you try to `(>):` append a list that doesn't exist already, you get an error!

`elements/components/cross/some-element.bst`:

```
kind: make

(@):
- include/cross-compile-component.yml

depends:
(>):
- another-element.bst

build-depends:
(>):
- components/host/python3.bst

...
```

So the issue is, while yes, I can see this is by design, there's a bit of a disconnect because the base-level `variables` YAML declaration syntax operates differently (maybe it's because it's an object in YAML and not a list?). You can specify items in `variables` in multiple files and it has no problem merging them as you would expect. This seems like it should be the same way `depends` should operate.

Is there some additional directive (maybe `(&):`) that could be added to declare a list as expandable/additive? I feel like this would make sense for "depends" list, since why would you ever want to completely redeclare your dependencies? (And you do have your `(=):` directive already to force-overwrite it should that need actually arise)

Additionally, just for convenience sake, the `(>):` append requires that that list was pre-defined. Maybe something like a `(?>):` directive could be added to append an existing list, or create it if it doesn't already exist? Something like Make's `?=` syntax would be nice to have, even for normal variables.

Let me know, or if I'm just doing something completely wrong :) Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the include examples in elements/components/cross/some-element.bst and include/cross-compile-component.yml, then locate BuildStream's YAML directive and list-merge handling. Done means the desired additive behavior is specified and covered for included depends and build-depends lists, including appending to a missing list; no test or implementation entry point is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.