bazelbuild / bazelbuild/bazel

Allow runfiles to be easily/efficiently passed as action inputs

Open
#15,486 19 comments 1 reaction 0 assignees View on GitHub
P4 team-Rules-API type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

If you have an action that needs to take **a collection of runfiles pertaining to multiple tools** as input, then it's very hard, error prone, and inefficient to correctly do this.

To correctly do this, the inputs passed to the action must include:
* `runfiles.files`.
* The `File`s within `runfiles.symlinks`. Getting these requires flattening the depset during analysis.
* The `File`s within `runfiles.root_symlinks`. Also requires flattening the depset during analysis.
* The symlink path strings of `runfiles.symlinks`. These either must be passed as args to the action, or a second file must be created to store their values and then _that_ file must be passed to the action.
* The symlink path strings of `runfiles.root_symlinks`. Same problem as above.
* The path strings of `runfiles.empty_filenames`. Accessing this attribute triggers execution of the underlying RunfilesSupplier, which is normally delayed until execution time. Also same problem as with the symlink paths.

The above is possible using pure-starlark, but is non-ideal because it has to flatten depsets (extract all the files and pass them as inputs; put the symlink paths and empty_filenames paths into Args objects).

Normally the answer to "i need runfiles passed to an action" is to use `tools` and pass FilesToRunProvider. However, that isn't always possible because:
1. FilesToRun can't be constructed by Starlark
2. Getting a reference to FilesToRun requires a target, i.e. a macro has to wire together multiple targets
3. (2) isn't always possible. Some examples: (a) if you want to properly implement `--stamp` behavior, embed build information, and trigger re-generation correctly, a target must pass all its runfiles et al to the action that generates that information; (b) for things like "build a zip file of myself" or some other packaging-esq type of thing.
5. FilesToRun is more expensive: it means "materialize this set of paths when the action is run", when the minimum necessary is "make sure any changes to the runfiles re-trigger the action execution" and/or "I just need the underlying inputs available and their symlink path-mapping" (e.g. a manifest)

From looking around, I think Java-native rules basically deal with this problem by using SourceManifestAction or middleman files[1]. The both work about the same: create a file and write the runfiles as a manifest, then have that file be an input to another action. All that work gets deferred until execution, time, though, and all analysis phase pays is passing the runfiles and registering an action.

So, a couple half-baked ideas:
1. Add a `runfiles_inputs` arg to `ctx.actions.run`. This would accept a list of runfiles objects. Like `inputs`, changes to them would re-trigger action execution. Whether they are materialized, and how they are, I don't know.
2. Expose some SourceManifestAction-equivalent, i.e. a way to correctly and efficiently transform a runfiles object into a file that can be passed to `inputs` and and `Args` objects.

(2) Seems better overall as it's more flexible. If you have a file, you can then pass it to `inputs` or add it to `args`.

[1] Small aside: middleman files don't seem to play well with Starlark implemented rules. From what I could tell, all Starlark rules call some RunfilesSupport helper logic during their Java-phase, which registers various actions and outputs. So interacting with runfiles support in the middle of the Starlark phase of a rule tends to create an action/output conflict with what happens in the Java phase.

### What underlying problem are you trying to solve with this feature?

Making it easy to correctly and efficiently pass runfiles information to actions.

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

google build

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

google build

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
n/a
```

### Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/15164 -- "i want to see the runfiles passed to an action"

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the ctx.actions.run entry point and compare the Java-native SourceManifestAction and middleman-file handling described in the issue. Examine RunfilesSupport and FilesToRunProvider to understand the existing constraints; the work is done when a precise, supported design for passing complete runfiles information to actions is implemented without analysis-time depset flattening.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.