apache / apache/beam

[Task][Go SDK]: Update datastoreio tests to not rely on direct runner.

Open
#27,549 1 comment 0 reactions 0 assignees View on GitHub
go io P3 stale task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

The tests rely on overriding the database client with a test interface double, however, it hasn't been done in a serializable form, relying on the Direct Runner to not serialize the DoFns.

eg. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/datastoreio/datastore.go#L44

This is not a good testing style as it isn't possible to execute them on portable runners, like prism, flink, spark, dataflow etc.

Since this requires a bit more thought and code to resolve, datastoreio will continue to use the direct runner as the default for now, but should be changed to an alternative testing means (possibly by a local GRPC connection or similar to a local fake of the service, setting the client endpoint to be overridden, with an API dialing option at client creation https://pkg.go.dev/google.golang.org/api/option#WithEndpoint)

This is used for the native SpannerIO tests, as an example. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/spannerio/common.go#L53

### Issue Priority

Priority: 3 (nice-to-have improvement)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.