bazel-contrib / bazel-contrib/bazel-lib

[FR]: Use bazel worker for copy_file_action

Open
#1,046 3 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted performance
Dominant language
Starlark
Stars
182
Forks
134
Avg merge
1d 46m
Merged PRs (30d)
1

Description

Hi, I [prototyped an idea](https://github.com/Andrius-B/bazel-lib/pull/1/files) about using a [bazel worker](https://bazel.build/remote/creating) and the naive implementation seems to copy the files over about 50% faster than using the `cp` process. I wanted to get some feedback if this kind of feature would be acceptable before spending more time on it (it's by no means ready for review yet).
This repository already has a tool written in `go` that copies directories `copy_directory`, so the toolchains and the release process are all in place. I would like to add a similar tool and related toolchains `copy_file`. Then in the `copy_file_action` this tool would be called instead of `cp` from `coreutils`.

The context for this change is that copying these files takes a while when there are a lot of source files. As an example of this case I created a small reproduction which generates `10_000` source `src/*.js` files and then builds a `js_binary` using all of those files, in tern calling `copy_file_action` on all of them. The whole build consists of only really copying files in this case so it's not a fair evaluation of how this feature would affect real builds, but gives some insights into the overhead of spawning so many processes.

With the latest `rules_js@2.1.3` which uses `cp` the build takes around 80 seconds on my M1 macbook:
```
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 83.830s, Critical Path: 3.58s
INFO: 10004 processes: 3 action cache hit, 3 internal, 10001 local.
INFO: Build completed successfully, 10004 total actions
bazel build example 0.06s user 0.06s system 0% cpu 1:25.07 total
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 83.391s, Critical Path: 3.64s
INFO: 10004 processes: 3 action cache hit, 3 internal, 10001 local.
INFO: Build completed successfully, 10004 total actions
bazel build example 0.06s user 0.08s system 0% cpu 1:25.63 total
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 82.971s, Critical Path: 3.68s
INFO: 10004 processes: 3 action cache hit, 3 internal, 10001 local.
INFO: Build completed successfully, 10004 total actions
bazel build example 0.06s user 0.07s system 0% cpu 1:24.75 total
```
Where as using a 4 worker processes inflates the action count 2x (as each copy actions gets an additional `WriteFile` action for the argument file), however it seems to run around twice as fast using a singleplex `proto` worker written in `go`. There are additional actions in this output since the build of the `copy_file` toolchain is cached as part of the build too:
```
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 40.807s, Critical Path: 4.89s
INFO: 20005 processes: 37 action cache hit, 10004 internal, 10001 worker.
INFO: Build completed successfully, 20005 total actions
bazel build example 0.05s user 0.06s system 0% cpu 43.757 total
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 37.610s, Critical Path: 4.96s
INFO: 20004 processes: 38 action cache hit, 10003 internal, 10001 worker.
INFO: Build completed successfully, 20004 total actions
bazel build example 0.04s user 0.05s system 0% cpu 39.256 total
$ python ./src/generate.py; time bazel build example
[..]
INFO: Elapsed time: 36.599s, Critical Path: 6.42s
INFO: 20004 processes: 4 action cache hit, 10003 internal, 10001 worker.
INFO: Build completed successfully, 20004 total actions
bazel build example 0.04s user 0.05s system 0% cpu 37.736 total
```

I am yet to test this on other platforms, but it looks promising. Let me know what you think

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Go copy_directory tool and the copy_file_action entry point, then review the proposed copy_file tool and related toolchains. Run the 10,000-file Python reproduction with Bazel to establish the current comparison. Done would mean a worker-backed copy_file implementation is integrated into copy_file_action and its behavior and performance are validated across platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.