apache / apache/pulsar

[dependency] The module `pulsar-zookeeper-utils` has potential conflict dependency problem

Open
#15,290 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
The Pulsar broker has a potential conflict dependency problem.

When building the KoP protocol handler, we met this conflict dependency problem. See error logs below:

```
[INFO] StreamNative :: Pulsar Protocol Handler :: Kafka Client Factory SUCCESS [ 0.196 s]
[INFO] StreamNative :: Pulsar Protocol Handler :: Kafka ... FAILURE [ 14.864 s]
[INFO] StreamNative :: Pulsar Protocol Handler :: OAuth 2.0 Client SKIPPED
[INFO] Kafka Payload Processor (Original) ................. SKIPPED
[INFO] StreamNative :: Pulsar Protocol Handler :: KoP Tests SKIPPED
[INFO] Kafka Payload Processor ............................ SKIPPED
[INFO] Tests for shaded kafka-payload-processor module .... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.702 s
[INFO] Finished at: 2022-04-23T22:05:31Z
[INFO] ------------------------------------------------------------------------
Error: Failed to execute goal on project pulsar-protocol-handler-kafka: Could not resolve dependencies for project io.streamnative.pulsar.handlers:pulsar-protocol-handler-kafka:jar:2.9.2.14: Failed to collect dependencies for io.streamnative.pulsar.handlers:pulsar-protocol-handler-kafka:jar:2.9.2.14: Could not resolve version conflict among [io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> org.apache.bookkeeper:stream-storage-server:jar:4.14.4 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.14.4 -> io.grpc:grpc-core:jar:1.42.1, io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> org.apache.bookkeeper:stream-storage-server:jar:4.14.4 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.14.4 -> io.grpc:grpc-testing:jar:1.42.1 -> io.grpc:grpc-core:jar:[1.42.1,1.42.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-grpclb:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-netty:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-rls:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-services:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1], io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 -> io.grpc:grpc-core:jar:1.45.1, io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 -> io.grpc:grpc-netty-shaded:jar:[1.45.1,1.45.1] -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]] -> [Help 1]
```

After format, we could find that the module `pulsar-zookeeper-utils` may depend on different versions package `grpc-core`.

```
Cannot resolve Could not resolve version conflict among

io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> org.apache.bookkeeper:stream-storage-server:jar:4.14.4 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.14.4 -> io.grpc:grpc-core:jar:1.42.1
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> org.apache.bookkeeper:stream-storage-server:jar:4.14.4 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.14.4 -> io.grpc:grpc-testing:jar:1.42.1 -> io.grpc:grpc-core:jar:[1.42.1,1.42.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-grpclb:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-netty:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-rls:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-services:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 -> io.grpc:grpc-core:jar:1.45.1
io.streamnative:pulsar-broker:jar:2.9.2.14 -> io.streamnative:pulsar-zookeeper-utils:jar:2.9.2.14 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-xds:jar:1.45.1 -> io.grpc:grpc-netty-shaded:jar:[1.45.1,1.45.1] -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]
```

I think we should resolve the dependency of the module `pulsar-zookeeper-utils`.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a new project
2. Depend on the package `io.streamnative:pulsar-broker:2.9.2.14`.
3. Try to build the project.
4. See error.

**Expected behavior**
The project should be built successfully.

**Context**
I think we could exclude all grpc-related packages from `org.apache.bookkeeper:stream-storage-java-client` in module `pulsar-zookeeper-utils` because it also import package `grpc-all`.

Contributor guide

Open the contributing guide

Research direction

Start in the pulsar-zookeeper-utils module's dependency declarations and trace its org.apache.bookkeeper:stream-storage-java-client and grpc-all dependencies. Reproduce the failure by depending on io.streamnative:pulsar-broker:2.9.2.14 and building a new project. Done means the conflicting grpc-core versions no longer prevent the project from building.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.