bazelbuild / bazelbuild/bazel

Add a possibility to lazily map a depset

Open
#13,132 10 comments 0 reactions 0 assignees View on GitHub
not stale P3 team-Rules-API type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

Hi dear Bazel folks,

My feature request is to add a possibility to apply a mapping function from F -> T onto a depset in order to lazily produce a depset of T from a depset of F. But of course I don't need this specific solution, I only need a solution for the problem I'm trying to solve.

### The problem I'm trying to solve?

In `rules_rust` we allow depending on rules providing `CcInfo` (provider used by C++ rules). `CcInfo` contains a depset of [linker inputs](https://docs.bazel.build/versions/master/skylark/lib/LinkerInput.html) through `cc_info.linking_context.linker_inputs`. Each linker input contains a collections of [LibraryToLink](https://docs.bazel.build/versions/master/skylark/lib/LibraryToLink.html) structs. We have to remember transitive LinkerInputs in Rust provider to generate correct linking command line, the simplified (very much simplified) algorithm is this:

```py
flags = []
for linker_input in linker_inputs.to_list():
for lib in linker_input.libraries_to_link:
if lib.alwayslink:
flags.extend([
"-C",
"link-arg=-Wl,--whole-archive",
"-C",
("link-arg=%s" % lib.static_library.path),
"-C",
"link-arg=-Wl,--no-whole-archive",
])
if lib.static_library:
flags.extend(["-lstatic=%s" % get_lib_name(lib.static_library)])
else:
flags.extend(["-ldylib=%s" % lib.dynamic_library)])
if linker_input.user_link_flags:
flags.extend(linker_input.user_link_flags)
```

The good news is that we can produce the command line here lazily (that means without having to iterate the depset in analysis) through `map_each` parameter on [`args.add_all`](https://docs.bazel.build/versions/master/skylark/lib/Args.html#add_all). `map_each` accepts a function that receives single parameter and returns a list of flags. This approach is safe to use from the build speed and memory use perspective, as the function cannot capture anything from its environment, it can only map the attribute to a return value.

**What we cannot do lazily is to collect input artifacts for the linking action**. We have to take the same depset of LinkerInputs struct from the example above and pick a specific library artifact roughtly following the same algorithm.

### Is there a workaround?

Yes, to flatten the nested set in analysis.

### Why do I need to solve this problem?

Without a solution it's not possible to depend on C++ rules from a Starlark rule in a memory and cpu efficient way. In addition, `rules_rust` have a very similar design challenge that C++ rules solved with LinkerInputs, and we would like to use the same solution.

### Why it hasn't been a problem yet?

Because C++ rules are implemented in Java where they can use `Iterables.transform`: https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/cpp/LinkerInputs.java;l=572?q=tolibraryartifacts. All existing Starlark rules that integrate with C++ rules (for example [`cc_shared_library`](https://github.com/bazelbuild/rules_cc/blob/master/examples/experimental_cc_shared_library.bzl#L220)) flatten the nested set.

### Why I think it's fine to map a depset?

In the past it was feared to call a user defined Starlark function in the execution phase. That fear is gone, we do exactly this in args. For `rules_rust` use case it's enough to impose exactly the same restrictions as on args - only functions with a single parameter are allowed.

CC @lberki @meisterT @oquenchil @comius

Thank you all and have a lovely day :)

Contributor guide

Open the contributing guide

Research direction

Start by reading the depset and Args.add_all APIs, then inspect src/main/java/com/google/devtools/build/lib/rules/cpp/LinkerInputs.java and the rules_cc example linked in the issue. Compare the existing lazy mapping restrictions with the rules_rust CcInfo use case. Done means an agreed, documented solution supports lazy artifact collection without analysis-time flattening.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.