apache / apache/hudi

Investigate: Dependency library sources are pulled during maven build

Open
#14,903 0 comments 0 reactions 0 assignees View on GitHub
area:build from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

Recently we encountered build slowness issue HUDI-2662 because of one of the dependent repositories was not reachable. Found that we are also downloading sources jar for several dependent libraries. The default maven build rule of skip sources and doc doesn't seem to work here. Tried explicitly updating the build scope, still didn't solve the issue. Need to investigate why sources jar are getting pulled for certain dependencies. 

 
{code:java}

[INFO] Including io.confluent:kafka-avro-serializer:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/kafka-avro-serializer/5.3.4/kafka-avro-serializer-5.3.4-sources.jar
[WARNING] Could not get sources for io.confluent:kafka-avro-serializer:jar:5.3.4:compile
[INFO] Including io.confluent:common-config:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/common-config/5.3.4/common-config-5.3.4-sources.jar
[WARNING] Could not get sources for io.confluent:common-config:jar:5.3.4:compile
[INFO] Including io.confluent:common-utils:jar:5.3.4 in the shaded jar.
Downloading from Maven Central: https://repo.maven.apache.org/maven2/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from confluent: https://packages.confluent.io/maven/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar
Downloading from pentaho.org: https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release/io/confluent/common-utils/5.3.4/common-utils-5.3.4-sources.jar

{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2664
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-2261

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Maven build configuration and reproduce the build while watching requests for sources JARs, especially for io.confluent:kafka-avro-serializer and its related dependencies. Trace which build or shading step requests them and compare that behavior with the configured source and documentation-skip settings. Done means the affected build no longer fetches unnecessary sources JARs and the build still produces the shaded artifact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.