apache / apache/beam

[Feature Request]: Use BOM to import com.google.apis dependencies instead of pinning a version

Open
#26,909 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

We currently have several dependencies pinned in [BeamModulePlugin.groovy](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L651-L656):

```
google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20220924-$google_clients_version",
google_api_services_cloudresourcemanager : "com.google.apis:google-api-services-cloudresourcemanager:v1-rev20220828-$google_clients_version",
google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20220920-$google_clients_version",
google_api_services_healthcare : "com.google.apis:google-api-services-healthcare:v1-rev20220818-$google_clients_version",
google_api_services_pubsub : "com.google.apis:google-api-services-pubsub:v1-rev20220904-$google_clients_version",
google_api_services_storage : "com.google.apis:google-api-services-storage:v1-rev20220705-$google_clients_version",
```

Even though we use Google Cloud's [libraries-bom](https://cloud.google.com/java/docs/bom) to rule specific versions.

Can we rely on the BOM instead of pinning specific versions?

This was recently the reason for a mismatch, and it might produce unpredictable behavior when using Beam, in case those APIs evolve.

For a more specific example, Beam 2.48.0 is pointing to `com.google.cloud:google-cloud-storage:2.22.1` and `google-cloud-storage:2.22.1` relies on `rev20230301`, but we pin `rev20220705`, which does not have some functionality.

For example, when `client.list(bucket, BlobListOption.prefix(prefix))` is used:

```
java.lang.NoSuchMethodError: 'com.google.api.services.storage.Storage$Objects$List com.google.api.services.storage.Storage$Objects$List.setMatchGlob(java.lang.String)'
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:426)
at com.google.cloud.storage.StorageImpl.lambda$listBlobs$11(StorageImpl.java:421)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:60)
at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:418)
at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:384)
```

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.