bazel-contrib / bazel-contrib/rules_jvm_external
race condition when fetching multiple unpinned repositories
- Dominant language
- Java
- Stars
- 373
- Forks
- 301
- Avg merge
- 7d 17h
- Merged PRs (30d)
- 3
Description
Our workspace has a tens of Maven artifact dependency trees. Each one of these is represented by a single `maven_install` repository rule, and there can be overlap between trees. When upgrading from rules_jvm_external 3.2 to 4.2, I'm encountering failures that I believe are caused by concurrent Coursier fetches for the same artifact.
Example:
```console
$ bazel run @unpinned_maven_deps//:pin
ERROR: An error occurred during the fetch of repository 'unpinned_maven_deps':
Traceback (most recent call last):
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 856, column 38, in _coursier_fetch_impl
dep_tree = make_coursier_dep_tree(
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 787, column 13, in make_coursier_dep_tree
fail("Error while fetching artifact with coursier: " + exec_result.stderr)
Error in fail: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar
ERROR: Error fetching repository: Traceback (most recent call last):
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 856, column 38, in _coursier_fetch_impl
dep_tree = make_coursier_dep_tree(
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 787, column 13, in make_coursier_dep_tree
fail("Error while fetching artifact with coursier: " + exec_result.stderr)
Error in fail: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar
ERROR: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar
```
On a different run w/ `RJE_VERBOSE=1`, I was able to obtain a different error message:
```
Error fetching artifacts:
https://../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar: download error: Caught java.nio.file.NoSuchFileException (.../.cache/coursier/v1/https/../org/eclipse/jdt/ecj/3.12.3/.ecj-3.12.3.jar.part -> .../.cache/coursier/v1/https/../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar) while downloading https://../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar
```
This second message leads me to https://github.com/coursier/coursier/issues/1815 .
So it seems that the shared/unsafe Coursier cache shouldn't be used by default, even in the unpinned case.
Contributor guide
Research direction
Start with coursier.bzl at _coursier_fetch_impl and make_coursier_dep_tree, then reproduce the failure with bazel run @unpinned_maven_deps//:pin and RJE_VERBOSE=1. Done means concurrent fetches for overlapping unpinned repositories no longer produce checksum or .part-file errors when resolving artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100