apache / apache/beam

GCP IO exposes protobuf on its API surface, causing user pain

Open
#18,476 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow gcp io java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in (at least) three versions of protobuf getting pulled in. These should be made to converge. We should consider using maven enforcer, which I think can check this.

```

[INFO] com.example:foo:jar:0.1
[INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
[INFO]
+- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
[INFO] | +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
[INFO]
| | \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - omitted for duplicate)
[INFO] | +-
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile
- omitted for conflict with 3.2.0)
[INFO] | | \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
[INFO]
| | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - omitted for conflict with 3.2.0)
[INFO]
| +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
[INFO] | | +- (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
- omitted for duplicate)
[INFO] | | +- (com.google.http-client:google-http-client:jar:1.20.0:compile
- omitted for conflict with 1.22.0)
[INFO] | | +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
[INFO]
| | | +- (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for conflict with
1.22.0)
[INFO] | | | \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - omitted for conflict
with 3.2.0)
[INFO] | +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
[INFO] |
| +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - omitted for conflict with 3.2.0)
[INFO]
| +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile
- omitted for duplicate)
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - omitted
for duplicate)

```

Incidentally, the dependency plugin stopped supporting the verbose tree, so we can't even visually inspect this except by downgrading.

Imported from Jira [BEAM-2523](https://issues.apache.org/jira/browse/BEAM-2523). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the dependency tree for the SDK, DataflowRunner, and GCP IO on one classpath, focusing on the conflicting protobuf versions shown in the report. Investigate whether Maven Enforcer can detect or prevent the mismatch; done means the relevant dependencies converge on a compatible protobuf version and the conflict can be checked reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.