Investigate: Dependency library sources are pulled during maven build
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
Recently we encountered build slowness issue HUDI-2662 because of one of the dependent repositories was not reachable. Found that we are also downloading sources jar for several dependent libraries. The default maven build rule of skip sources and doc doesn't seem to work here. Tried explicitly updating the build scope, still didn't solve the issue. Need to investigate why sources jar are getting pulled for certain dependencies.
{code:java}
[INFO] Including io.confluent:kafka-avro-serializer:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
[WARNING] Could not get sources for io.confluent:kafka-avro-serializer:jar:5.3.4:compile
[INFO] Including io.confluent:common-config:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
[WARNING] Could not get sources for io.confluent:common-config:jar:5.3.4:compile
[INFO] Including io.confluent:common-utils:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
{code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2664
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-2261
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Maven build configuration and reproduce the build while watching requests for sources JARs, especially for io.confluent:kafka-avro-serializer and its related dependencies. Trace which build or shading step requests them and compare that behavior with the configured source and documentation-skip settings. Done means the affected build no longer fetches unnecessary sources JARs and the build still produces the shaded artifact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100