bazel-contrib / bazel-contrib/rules_jvm_external

race condition when fetching multiple unpinned repositories

Open
#640 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
373
Forks
301
Avg merge
7d 17h
Merged PRs (30d)
3

Description

Our workspace has a tens of Maven artifact dependency trees. Each one of these is represented by a single `maven_install` repository rule, and there can be overlap between trees. When upgrading from rules_jvm_external 3.2 to 4.2, I'm encountering failures that I believe are caused by concurrent Coursier fetches for the same artifact.

Example:

```console
$ bazel run @unpinned_maven_deps//:pin
ERROR: An error occurred during the fetch of repository 'unpinned_maven_deps':
Traceback (most recent call last):
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 856, column 38, in _coursier_fetch_impl
dep_tree = make_coursier_dep_tree(
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 787, column 13, in make_coursier_dep_tree
fail("Error while fetching artifact with coursier: " + exec_result.stderr)
Error in fail: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar

ERROR: Error fetching repository: Traceback (most recent call last):
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 856, column 38, in _coursier_fetch_impl
dep_tree = make_coursier_dep_tree(
File ".../d28a7546cffbbfc2d184703c8d8422c1/external/rules_jvm_external/coursier.bzl", line 787, column 13, in make_coursier_dep_tree
fail("Error while fetching artifact with coursier: " + exec_result.stderr)
Error in fail: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar

ERROR: Error while fetching artifact with coursier: Error fetching artifacts:
https://../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar: checksum not found: .../.cache/coursier/v1/https/../io/netty/netty-handler-proxy/4.1.48.Final/netty-handler-proxy-4.1.48.Final.jar
```

On a different run w/ `RJE_VERBOSE=1`, I was able to obtain a different error message:

```
Error fetching artifacts:
https://../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar: download error: Caught java.nio.file.NoSuchFileException (.../.cache/coursier/v1/https/../org/eclipse/jdt/ecj/3.12.3/.ecj-3.12.3.jar.part -> .../.cache/coursier/v1/https/../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar) while downloading https://../org/eclipse/jdt/ecj/3.12.3/ecj-3.12.3.jar
```

This second message leads me to https://github.com/coursier/coursier/issues/1815 .

So it seems that the shared/unsafe Coursier cache shouldn't be used by default, even in the unpinned case.

Contributor guide

Open the contributing guide

Research direction

Start with coursier.bzl at _coursier_fetch_impl and make_coursier_dep_tree, then reproduce the failure with bazel run @unpinned_maven_deps//:pin and RJE_VERBOSE=1. Done means concurrent fetches for overlapping unpinned repositories no longer produce checksum or .part-file errors when resolving artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.