bazelbuild / bazelbuild/rules_rust
Improve cargo splicing and lock file generation in large projects
- Dominant language
- Starlark
- Stars
- 843
- Forks
- 651
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 15
Description
In https://github.com/bazelbuild/rules_rust/pull/3467, I made some experiments about when cargo splicing is called and which lock file can help to reduce this.
## New Problem
Following the recommended approach of using a `MODULE.bazel.lock` on a large scale rust project shows a new issue. The `MODULE.bazel.lock` file stores the shasum of the `Cargo.lock` file because it is used in the following setup:
```starlark
crate.from_cargo(
cargo_lockfile = "//:Cargo.lock",
manifests = [
"//:Cargo.toml",
# many more other Cargo.toml files
],
# ...
)
```
This makes the `MODULE.bazel.lock` file practically unusable due to a high number of merge conflicts.
Any time a developer changes a dependency between internal `Cargo.toml` files, the `Cargo.lock` file is changed causing the shasum in the `MODULE.bazel.lock` file to change.
## The proposed solution
Based on a discussion with @illicitonion, we identified that rules_rust does not really care about the internal cargo dependencies, but only about the external/third party dependencies.
For this, we wanted to introduce an intermediate lockfile that only contains the external dependencies.
```starlark
crate.from_cargo(
cargo_lockfile = "//:Cargo.bazel.lock", # no internal deps anymore, just third party
manifests = [
"//:Cargo.toml",
# many more other Cargo.toml files
],
# ...
)
```
The `Cargo.bazel.lock` file is generated from the `Cargo.lock` file by removing all internal dependencies and only keeping the external dependencies.
Having only the external dependencies in the `Cargo.bazel.lock` file allows to avoid merge conflicts when changing internal dependencies.
The shasum of the `Cargo.bazel.lock` file in the `MODULE.bazel.lock` will only change if the external dependencies change.
This happens usually on a much smaller frequency than the internal dependencies.
In essence, this can be done with
```python
cargo_lock_file = parse(Path("Cargo.lock").read_text())
cargo_lock_file["package"] = [
package for package in cargo_lock_file["package"] if "source" in package
]
Path("Cargo.bazel.lock").write_text(dumps(cargo_lock_file))
```
## Observation from the new solution
Trying this new solution did not work as expected.
After passing the `Cargo.bazel.lock` file to `crate.from_cargo`, splicing runs as expected.
However, rules_rust fully restores the content of the `Cargo.lock` file into the `Cargo.bazel.lock` file.
## Summary of this problem as BazelCon talk
For people landing here who want to get an overview, take a look at [this BazelCon talk](https://www.youtube.com/watch?v=IvJD0kSJcoI).
Contributor guide
Assessment
This issue has not been assessed yet.