Monolithic MODULE.bazel.lock scales/interacts poorly with lockfile facts
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
https://github.com/hermeticbuild/rules_rs makes heavy use of the lockfile facts API to perform Cargo crate resolution and avoid registry metadata fetches. (Specifically, the existing Cargo.lock records the set of all possible dependencies for each crate along with their hashes, but in order to compute the actual set of dependencies, taking into account optional and platform-specific deps, each crate's metadata must be fetched from the registry. rules_rs records the minimal amount of extra information about the crate's own features and the flavor of each dependency that is needed to perform the correct resolution).
rules_rs already performs dedupe of all facts across all the transitive closures, but given Rust transitive closures do not enforce a single version, it is common to end up with multiple versions of each crate and it's metadata. While there are individual crates that do unusual things (like `web-sys` defining a feature for seemingly every WebIDL symbol), the vast majority of crates have <5 features and <10 dependencies.
All of this combines to a state where for our (fairly large, but not out of the ordinary) repo, the MODULE.bazel.lock is around 4MB. This can cause issues because Github places limits on file size, and even if those limits are lifted, frequent churn of a single large text file leads to packfile bloat and git performance issues.
The lockfile facts do allow extremely fast resolution though (we can resolve in <100ms whereas rules_rust took minutes) and allow for ergonomic airgapped builds.
It would be ideal if there was a way to shard this lockfile or some other equivalent approach that allowed us to records the set of facts we need without this file size hazard.
### Which category does this issue belong to?
External Dependency
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
This is not our actual repo, but it's a representative one:
https://github.com/hermeticbuild/rules_rs/blob/main/test/MODULE.bazel
https://raw.githubusercontent.com/hermeticbuild/rules_rs/refs/heads/main/test/MODULE.bazel.lock
### Which operating system are you running Bazel on?
OSX
### What is the output of `bazel info release`?
9.1.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with test/MODULE.bazel and the linked test/MODULE.bazel.lock to understand how the representative repository records lockfile facts and how large the result becomes. Trace the lockfile facts API and related external-dependency handling, then define a sharding or equivalent design that preserves fast resolution and airgapped builds without concentrating all facts in one large file.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100