bazelbuild / bazelbuild/bazel

FR: allow genrules to produce directories (tree artifacts)

Open
#19,030 7 comments 3 reactions 0 assignees View on GitHub
not stale P4 team-Rules-API type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

It'd be cool if Bazel officially supported directories as first-class outputs of genrules. If including directories in `outs` is undesirable for some reason, perhaps they could be declared using a separate `outdirs` attribute, each of which would roughly map to a regular rule's `ctx.actions.declare_directory`.

Alternatively, would a Starlark impl of genrule (w/ separate `outdirs` semantics) in Skylib make sense? (We don't have to worry about Windows users yet, but how would cmd_bat/cmd_ps map to ctx.actions.run/run_shell? Would there be a separate FR for run_bat+run_ps?)

### What underlying problem are you trying to solve with this feature?

I have to support novice Bazel users who are comfortable writing genrules but are intimidated by Bazel's rule implementations, especially when they only need a rule for a single target. (I.e. there is no need for reuse.) These users are not build engineers, so they don't want to invest time in learning about Bazel's execution model, constraints for RBE, etc. On their own, genrule's `srcs`, `outs`, and `cmd` are easy enough for most folks to understand. At the same time, I can't be the SPOF the instant they need to update one of their genrules to produce a directory instead of a file.

To date, we've been in the habit of exporting directories as zips (using a custom wrapper around Bazel's `zipper` for reproducibility), and then unzipping them somewhere else. This is clumsy when feeding outputs to `@rules_pkg//pkg:mappings.bzl%pkg_files`, since we have to

1. generate a directory tree (this is the bulk of the genrule's `cmd`);
2. zip it w/ custom tool (boilerplate appearing at end of `cmd`);
3. unzip it (and assign it a target name) to a TreeArtifact using an internal `unzip` rule; and then
4. feed that TreeArtifact to `pkg_files.srcs`.

I'd rather send a TreeArtifact directly from a genrule to pkg_files.

The most common use case for this is any sort of generated code or documentation. This seems like a common use case across Bazel's entire userbase (e.g. issue #1025).

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

release 6.0.0-vmware

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

I reviewed GitHub issues by searching for "in:title genrule". No relevant hits.

Other relevant refs:

* https://bazel.build/reference/be/general#general-advice
* "its dependency checking of directories is unsound"
* https://github.com/bazelbuild/bazel/issues/1025
* https://github.com/bazelbuild/bazel/issues/12781
* https://github.com/bazelbuild/bazel/issues/18646
* prompted by https://github.com/bazelbuild/bazel/issues/18579#issuecomment-1577228250 ?

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with Bazel's genrule semantics and the referenced issues #1025, #12781, and #18646; compare the requested outdirs behavior with ctx.actions.declare_directory and rules_pkg's pkg_files.srcs use case. Done means the project has an agreed approach for direct TreeArtifact outputs, including the cmd_bat/cmd_ps and Skylib alternatives.

Written by the indexing model from the issue text.

Assessment

Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.