apache / apache/beam

[Feature Request][Go SDK]: Have the Dataflow runner use the cloned GCR images instead if the image is unset.

Open
#23,453 6 comments 0 reactions 0 assignees View on GitHub
dataflow go new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Java and Python SDKs use cloned external images from Dockerhub cached in GCR by default instead.

https://github.com/apache/beam/search?q=v1beta3%2F

Containers live here:
https://console.cloud.google.com/gcr/images/cloud-dataflow/global/v1beta3

The Go SDK should do the same, to reduce startup time.
Container paths would be like `gcr.io/cloud-dataflow/v1beta3/beam_go_sdk:2.37.0` and similar for each version.

The Go SDK container is significantly smaller than Java and Python's due to the static compilation nature of Go, so this likely reduced issues requiring the proximity caching. However, for parity this would be useful.

~~Images are also hosted on Google Artifact Registry on a Regional basis:
eg.
https://console.cloud.google.com/artifacts/docker/google.com:dataflow-containers/us-west1/worker/v1beta3%2Fbeam_go_sdk?project=google.com:dataflow-containers

With paths like `us-west1-docker.pkg.dev/google.com/dataflow-containers/worker/v1beta3/beam_go_sdk:2.42.0`

But this also appears to require certain additional gcloud authentications to work 100%, so unless dataflow service accounts/workers are automatically authenticated for fetching images, it may not yet be worthwhile to implement this.~~

GAR containers are substituted by the dataflow service assuming the GCR paths are used.

### Issue Priority

Priority: 3

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.