bazelbuild / bazelbuild/bazel

Parallelize repository up-to-dateness check

Open
#20,869 2 comments 0 reactions 1 assignee Claimed by @meteorcloudy View on GitHub
P2 team-ExternalDeps type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

External repositories need to verify that their file dependencies are still up-to-date. This happens on one thread in `RepositoryFunction.verifyMarkerDataForFiles()`. This means that if a repository has many files to check, we are leaving a lot of performance on the table. It would be nice if we used all the parallelism available (it's mostly checksumming, so it's not obvious whether it's I/O or CPU-bound, but more parallelism helps either way, the question is just how much more)

The obvious way to do this is to have a little thread pool, but we already have many little thread pools (e.g. `TreeArtifactValue.VISITOR_POOL` which does something similar), so maybe some consolidation is in order.

### Which category does this issue belong to?

_No response_

### What underlying problem are you trying to solve with this feature?

Make null builds faster.

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
`7dd569f1b0166cbf651789d91ce192d8bde44267`
```

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.