pinot-clients have too many dependencies
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
## Problem
Since 0.11.0, `pinot-jdbc-client` and pinot `pinot-java-client` have a dependency on `pinot-common`.
See introductions:
https://github.com/apache/pinot/commit/15e939818ca913bd9a5f0162300057c1a8b20f39
https://github.com/apache/pinot/commit/23a81d07b52dad6181b696562e07cdfa0932d191
This introduces the following new dependencies to the clients:
```
\- org.apache.pinot:pinot-common:jar:0.12.1:compile
+- org.apache.httpcomponents:httpmime:jar:4.5.13:compile
+- org.antlr:antlr4-runtime:jar:4.6:compile
+- org.apache.thrift:libthrift:jar:0.15.0:compile
| \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
+- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile
+- com.github.luben:zstd-jni:jar:1.5.2-3:compile
+- org.lz4:lz4-java:jar:1.8.0:compile
+- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:compile
| \- org.apache.logging.log4j:log4j-core:jar:2.17.1:runtime
+- commons-httpclient:commons-httpclient:jar:3.1:compile
+- it.unimi.dsi:fastutil:jar:8.2.3:compile
+- org.webjars:swagger-ui:jar:3.23.11:compile
+- io.grpc:grpc-netty-shaded:jar:1.41.0:compile
| +- io.perfmark:perfmark-api:jar:0.23.0:runtime
| \- io.grpc:grpc-core:jar:1.41.0:compile
| +- com.google.code.gson:gson:jar:2.8.6:runtime
| +- com.google.android:annotations:jar:4.1.1.4:runtime
| \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.19:runtime
+- io.grpc:grpc-protobuf:jar:1.41.0:compile
| +- io.grpc:grpc-api:jar:1.41.0:compile
| | \- io.grpc:grpc-context:jar:1.41.0:compile
| +- com.google.api.grpc:proto-google-common-protos:jar:2.0.1:compile
| \- io.grpc:grpc-protobuf-lite:jar:1.41.0:compile
+- io.grpc:grpc-stub:jar:1.41.0:compile
+- org.apache.yetus:audience-annotations:jar:0.13.0:compile
+- org.mindrot:jbcrypt:jar:0.4:compile
\- com.github.seancfoley:ipaddress:jar:5.3.4:compile
```
This is a lot of new, unused dependencies for the client.
Also notice that an implementation of log4j is included `org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:compile`, which is bad practice but easy to quickfix with an exclusion in the pom.
Then for `pinot-jdbc-client` a dependency on `pinot-core` was introduced:
https://github.com/apache/pinot/commit/caf8d755820d6bf27bc7daeb74ac35d2e70caa61
This introduces the following dependencies in the client
```
+- org.apache.pinot:pinot-jdbc-client:jar:0.12.1:compile
| +- org.apache.pinot:pinot-core:jar:0.12.1:compile
| | +- com.uber:h3:jar:4.0.0:compile
| | +- org.roaringbitmap:RoaringBitmap:jar:0.9.35:compile
| | | \- org.roaringbitmap:shims:jar:0.9.35:runtime
| | +- org.apache.pinot:pinot-segment-spi:jar:0.12.1:compile
| | +- org.apache.pinot:pinot-segment-local:jar:0.12.1:compile
| | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.79.Final:compile
| | | +- io.netty:netty-common:jar:4.1.79.Final:compile
| | | +- io.netty:netty-buffer:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport-native-unix-common:jar:4.1.79.Final:compile
| | | \- io.netty:netty-transport-classes-epoll:jar:4.1.79.Final:compile
| | +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.79.Final:compile
| | | \- io.netty:netty-transport-classes-kqueue:jar:4.1.79.Final:compile
| | +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.53.Final:compile
| | | +- io.netty:netty-tcnative-classes:jar:2.0.53.Final:compile
| | | +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.53.Final:compile
| | | +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.53.Final:compile
| | | \- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.53.Final:compile
| | +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.53.Final:compile
| | +- io.netty:netty-all:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-dns:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-haproxy:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-http2:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-memcache:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-mqtt:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-redis:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-smtp:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-stomp:jar:4.1.79.Final:compile
| | | +- io.netty:netty-codec-xml:jar:4.1.79.Final:compile
| | | +- io.netty:netty-resolver:jar:4.1.79.Final:compile
| | | +- io.netty:netty-resolver-dns:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport-rxtx:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport-sctp:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport-udt:jar:4.1.79.Final:compile
| | | +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.79.Final:compile
| | | +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.79.Final:runtime
| | | +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.79.Final:runtime
| | | +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.79.Final:runtime
| | | \- io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.79.Final:runtime
| | +- com.clearspring.analytics:stream:jar:2.7.0:compile
| | +- org.apache.datasketches:datasketches-java:jar:1.2.0-incubating:compile
| | | \- org.apache.datasketches:datasketches-memory:jar:1.2.0-incubating:compile
| | +- com.tdunning:t-digest:jar:3.2:compile
| | +- org.xerial.larray:larray-mmap:jar:0.4.1:compile
| | | \- org.xerial.larray:larray-buffer:jar:0.4.1:compile
| | +- net.sf.jopt-simple:jopt-simple:jar:4.6:compile
| | +- org.glassfish.jersey.containers:jersey-container-grizzly2-http:jar:2.35:compile
| | | \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:compile
| | +- org.glassfish.grizzly:grizzly-http-server:jar:2.4.4:compile
| | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:compile
| | +- org.apache.lucene:lucene-core:jar:8.2.0:compile
| | +- org.apache.lucene:lucene-queryparser:jar:8.2.0:compile
| | | +- org.apache.lucene:lucene-queries:jar:8.2.0:compile
| | | \- org.apache.lucene:lucene-sandbox:jar:8.2.0:compile
| | \- org.apache.lucene:lucene-analyzers-common:jar:8.2.0:compile
| +- org.asynchttpclient:async-http-client:jar:2.12.3:compile
| | +- org.asynchttpclient:async-http-client-netty-utils:jar:2.12.3:compile
| | +- io.netty:netty-codec-http:jar:4.1.60.Final:compile
| | +- io.netty:netty-handler:jar:4.1.60.Final:compile
| | +- io.netty:netty-codec-socks:jar:4.1.60.Final:compile
| | +- io.netty:netty-handler-proxy:jar:4.1.60.Final:compile
| | +- org.reactivestreams:reactive-streams:jar:1.0.3:compile
| | +- com.typesafe.netty:netty-reactive-streams:jar:2.0.4:compile
| | \- com.sun.activation:jakarta.activation:jar:1.2.2:compile
| \- com.101tec:zkclient:jar:0.7:compile
```
Impact:
The pinot clients are too heavy and introduce too many unused dependencies that can conflict with the users's project. We are stuck on client `0.10.0`.
## Suggestion
From what I understand of the commits the goal of importing pinot-common and pinot-core was to get access to a few utils, and almost all of the dependencies are not used.
I'd be nice to refactor dependencies to make the pinot clients light again.
Contributor guide
Research direction
Start by inspecting the dependency declarations for pinot-jdbc-client and pinot-java-client and compare them with the dependency trees shown in the issue. Trace which utilities require pinot-common and pinot-core, then refactor the client dependencies so unused transitive libraries and the Log4j implementation are no longer included; verify that the clients remain usable with a substantially lighter dependency tree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100