apache / apache/maven-shade-plugin

[MSHADE-329] Concurrent writes of dependency-reduced-pom.xml adds same exclusion ad infinitum

Open
#328 1 comment 0 reactions 0 assignees View on GitHub
bug priority:major
Dominant language
Java
Stars
188
Forks
103
Avg merge
16h 39m
Merged PRs (30d)
4

Description

**[Håkon Hallingstad](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hakonhall)** opened **[MSHADE-329](https://issues.apache.org/jira/browse/MSHADE-329?redirect=false)** and commented

I have a multi-threaded `mvn install` that seems to halt but ends up using 200% CPU, in about 50% of the invocations. The `mvn` command used is:

```
mvn -T1C -nsu -Dmaven.source.skip -Dmaven.javadoc.skip -Dmaven.test.skip install -rf :MODULE
```

Using `mvnDebug` I have found out that there are 2 running Java threads, each writing `dependency-reduced-pom.xml` in two different modules `A` and `B`, respectively. These files seems to become several MB large, before they're deleted and then written again, and so forth.

I have looked into one of the threads, and there is a `rewriteDependencyReducedPomIfWeHaveReduction` in ShadeMojo with a `loopCounter` with value 2735, that just keeps increasing. Presumably there is something like one dependency-reduced-pom.xml written per iteration.

From the source code it seems this can only happen if https://github.com/apache/maven-shade-plugin/blob/master/src/main/java/org/apache/maven/plugins/shade/mojo/ShadeMojo.java#L1172 is hit at least that number of times, meaning the ShadeMojo adds that many exclusions, which seems to correspond to hamcrest-core:

```

 
    junit
    junit
     4.12
    test
   
     
        hamcrest-core
        org.hamcrest
     
     
        hamcrest-core
        org.hamcrest
     
     ...
```

```
grep hamcrest-core dependency-reduced-pom.xml | wc -l
2735
```

 

The same exclusion is added `loopCounter` times in `updateExcludesInDeps`.

There is no problem running `mvn` in single-thread mode.

---

**Affects:** 3.2.1

**Attachments:**
- [mshade-329-thread-dump.txt](https://issues.apache.org/jira/secure/attachment/13018805/mshade-329-thread-dump.txt) (_34.15 kB_)

**Issue Links:**
- [MSHADE-124](https://issues.apache.org/jira/browse/MSHADE-124) Need better plan for getting dependency-reduced-pom.xml out of basedir

- [FLINK-14318](https://issues.apache.org/jira/browse/FLINK-14318) JDK11 build stalls during shading

1 votes, 5 watchers

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ShadeMojo.java, especially rewriteDependencyReducedPomIfWeHaveReduction and updateExcludesInDeps, then reproduce with the reported parallel mvn install command or inspect mshade-329-thread-dump.txt. Check how concurrent writes to dependency-reduced-pom.xml cause repeated hamcrest-core exclusions; done means the parallel build no longer loops or produces unbounded duplicate exclusions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.