apache / apache/maven-shade-plugin
[MSHADE-329] Concurrent writes of dependency-reduced-pom.xml adds same exclusion ad infinitum
- Dominant language
- Java
- Stars
- 188
- Forks
- 103
- Avg merge
- 16h 39m
- Merged PRs (30d)
- 4
Description
**[Håkon Hallingstad](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hakonhall)** opened **[MSHADE-329](https://issues.apache.org/jira/browse/MSHADE-329?redirect=false)** and commented
I have a multi-threaded `mvn install` that seems to halt but ends up using 200% CPU, in about 50% of the invocations. The `mvn` command used is:
```
mvn -T1C -nsu -Dmaven.source.skip -Dmaven.javadoc.skip -Dmaven.test.skip install -rf :MODULE
```
Using `mvnDebug` I have found out that there are 2 running Java threads, each writing `dependency-reduced-pom.xml` in two different modules `A` and `B`, respectively. These files seems to become several MB large, before they're deleted and then written again, and so forth.
I have looked into one of the threads, and there is a `rewriteDependencyReducedPomIfWeHaveReduction` in ShadeMojo with a `loopCounter` with value 2735, that just keeps increasing. Presumably there is something like one dependency-reduced-pom.xml written per iteration.
From the source code it seems this can only happen if https://github.com/apache/maven-shade-plugin/blob/master/src/main/java/org/apache/maven/plugins/shade/mojo/ShadeMojo.java#L1172 is hit at least that number of times, meaning the ShadeMojo adds that many exclusions, which seems to correspond to hamcrest-core:
```
junit
junit
4.12
test
hamcrest-core
org.hamcrest
hamcrest-core
org.hamcrest
...
```
```
grep hamcrest-core dependency-reduced-pom.xml | wc -l
2735
```
The same exclusion is added `loopCounter` times in `updateExcludesInDeps`.
There is no problem running `mvn` in single-thread mode.
---
**Affects:** 3.2.1
**Attachments:**
- [mshade-329-thread-dump.txt](https://issues.apache.org/jira/secure/attachment/13018805/mshade-329-thread-dump.txt) (_34.15 kB_)
**Issue Links:**
- [MSHADE-124](https://issues.apache.org/jira/browse/MSHADE-124) Need better plan for getting dependency-reduced-pom.xml out of basedir
- [FLINK-14318](https://issues.apache.org/jira/browse/FLINK-14318) JDK11 build stalls during shading
1 votes, 5 watchers
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ShadeMojo.java, especially rewriteDependencyReducedPomIfWeHaveReduction and updateExcludesInDeps, then reproduce with the reported parallel mvn install command or inspect mshade-329-thread-dump.txt. Check how concurrent writes to dependency-reduced-pom.xml cause repeated hamcrest-core exclusions; done means the parallel build no longer loops or produces unbounded duplicate exclusions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100