bazelbuild / bazelbuild/rules_rust

Splicing a workspace is slow

Open
#1,873 3 comments 3 reactions 0 assignees View on GitHub
needs-triage
Dominant language
Starlark
Stars
843
Forks
651
Avg merge
2d 18h
Merged PRs (30d)
15

Description

Sample repo: https://github.com/typesanitizer/slow-repin-lib

`CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index` takes over 2-3 minutes to complete on my M1 Mac, for 169 dependencies. In contrast, a clean `cargo build`, including fetching dependencies, takes about 1m15s.

This example repo is somewhat simplified from our monorepo https://github.com/sourcegraph/sourcegraph, where it takes 150%-250% more time, with ~40% more deps in total.

I understand that some of the overhead may be due to sandboxing on macOS, but I suspect that's probably not the full explanation (it looks like `--sandbox_strategy=local` doesn't work for `bazel sync`, so I couldn't tell how to test without sandboxing).

I don't know if there is a standard way of printing the executed commands (`--subcommands` doesn't work with `bazel sync`), so I hackily added a `print` invocation before the `cargo-bazel` invocation in a local checkout of `rules_rust` and got something like:

```CARGO=/private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/bin/cargo RUSTC=/private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/bin/rustc RUST_BACKTRACE=full CARGO_HOME=/private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/crate_index/.cargo_home /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/crate_index/cargo-bazel splice --output-dir /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/crate_index/splicing-output --splicing-manifest /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/crate_index/splicing_manifest.json --config /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/crate_index/cargo-bazel.json --cargo /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/bin/cargo --rustc /private/var/tmp/_bazel_varun/9733e4cbb184d6ba5c16ee75a5b939da/external/rust_darwin_aarch64__aarch64-apple-darwin__stable_tools/bin/rustc --cargo-lockfile /Users/varun/Code/issues/slow-repin/slow-repin-lib/Cargo.lock```

On re-running that command with cargo flamegraph (I'm not sure if I need to be aware of extra caches?), I got this flamegraph: (GitHub doesn't provide interactivity, but you should be able to see more information if you download it from https://user-images.githubusercontent.com/7187503/224541869-3dc2f87e-bbd3-41c7-bcc2-c6c112d5a90a.svg and open it in the browser)

![flamegraph](https://user-images.githubusercontent.com/7187503/224541869-3dc2f87e-bbd3-41c7-bcc2-c6c112d5a90a.svg)

Is there a way to speed up the splicing step on re-pinning?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.