GCP IO exposes protobuf on its API surface, causing user pain
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in (at least) three versions of protobuf getting pulled in. These should be made to converge. We should consider using maven enforcer, which I think can check this.
```
[INFO] com.example:foo:jar:0.1
[INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
[INFO]
+- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
[INFO] | +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
[INFO]
| | \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - omitted for duplicate)
[INFO] | +-
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile
- omitted for conflict with 3.2.0)
[INFO] | | \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
[INFO]
| | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - omitted for conflict with 3.2.0)
[INFO]
| +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
[INFO] | | +- (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
- omitted for duplicate)
[INFO] | | +- (com.google.http-client:google-http-client:jar:1.20.0:compile
- omitted for conflict with 1.22.0)
[INFO] | | +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
[INFO]
| | | +- (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for conflict with
1.22.0)
[INFO] | | | \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - omitted for conflict
with 3.2.0)
[INFO] | +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
[INFO] |
| +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - omitted for conflict with 3.2.0)
[INFO]
| +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile
- omitted for duplicate)
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - omitted
for duplicate)
```
Incidentally, the dependency plugin stopped supporting the verbose tree, so we can't even visually inspect this except by downgrading.
Imported from Jira [BEAM-2523](https://issues.apache.org/jira/browse/BEAM-2523). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Research direction
Start by reproducing the dependency tree for the SDK, DataflowRunner, and GCP IO on one classpath, focusing on the conflicting protobuf versions shown in the report. Investigate whether Maven Enforcer can detect or prevent the mismatch; done means the relevant dependencies converge on a compatible protobuf version and the conflict can be checked reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100