bazelbuild / bazelbuild/rules_rust

Improve cargo splicing and lock file generation in large projects

Open
#3,522 6 comments 0 reactions 0 assignees View on GitHub
crate-universe
Dominant language
Starlark
Stars
843
Forks
651
Avg merge
2d 18h
Merged PRs (30d)
15

Description

In https://github.com/bazelbuild/rules_rust/pull/3467, I made some experiments about when cargo splicing is called and which lock file can help to reduce this.

## New Problem

Following the recommended approach of using a `MODULE.bazel.lock` on a large scale rust project shows a new issue. The `MODULE.bazel.lock` file stores the shasum of the `Cargo.lock` file because it is used in the following setup:

```starlark
crate.from_cargo(
cargo_lockfile = "//:Cargo.lock",
manifests = [
"//:Cargo.toml",
# many more other Cargo.toml files
],
# ...
)
```

This makes the `MODULE.bazel.lock` file practically unusable due to a high number of merge conflicts.
Any time a developer changes a dependency between internal `Cargo.toml` files, the `Cargo.lock` file is changed causing the shasum in the `MODULE.bazel.lock` file to change.

## The proposed solution

Based on a discussion with @illicitonion, we identified that rules_rust does not really care about the internal cargo dependencies, but only about the external/third party dependencies.
For this, we wanted to introduce an intermediate lockfile that only contains the external dependencies.

```starlark
crate.from_cargo(
cargo_lockfile = "//:Cargo.bazel.lock", # no internal deps anymore, just third party
manifests = [
"//:Cargo.toml",
# many more other Cargo.toml files
],
# ...
)
```

The `Cargo.bazel.lock` file is generated from the `Cargo.lock` file by removing all internal dependencies and only keeping the external dependencies.
Having only the external dependencies in the `Cargo.bazel.lock` file allows to avoid merge conflicts when changing internal dependencies.
The shasum of the `Cargo.bazel.lock` file in the `MODULE.bazel.lock` will only change if the external dependencies change.
This happens usually on a much smaller frequency than the internal dependencies.

In essence, this can be done with

```python
cargo_lock_file = parse(Path("Cargo.lock").read_text())
cargo_lock_file["package"] = [
package for package in cargo_lock_file["package"] if "source" in package
]
Path("Cargo.bazel.lock").write_text(dumps(cargo_lock_file))
```

## Observation from the new solution

Trying this new solution did not work as expected.
After passing the `Cargo.bazel.lock` file to `crate.from_cargo`, splicing runs as expected.
However, rules_rust fully restores the content of the `Cargo.lock` file into the `Cargo.bazel.lock` file.

## Summary of this problem as BazelCon talk

For people landing here who want to get an overview, take a look at [this BazelCon talk](https://www.youtube.com/watch?v=IvJD0kSJcoI).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.