apache / apache/beam

[New Feature]: Be able to Pass additional args to automated expansion service startup [Go SDK]

Open
#22,302 0 comments 0 reactions 0 assignees View on GitHub
cross-language go new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?
Edit: Changed from Bug to New Feature
We should be able to plumb additional flags to the automatically started up Java Expansion service, in case there are specific flags IOs need. Eg. for a recent KafkaIO issue that didn't appear to be able to worked around centrally. What follows is the original bug and proposal.

The Java KafkaIO SDF implementation is experiencing issues #21730 and the Xlang wrappers need to be updated to pass an experiment flag to the Java Expansion service.

This issue tracks doing the same to the Go KafkaIO wrapper, to pass the additional args to the automatically started up expansion service, as it was done to the python wrapper in #22286.

`--experiments=use_unbounded_sdf_wrapper` in particular.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/xlang/kafkaio/kafka.go#L90

Additional args can probably be added as a new "ExpansionServiceOption"
https://github.com/apache/beam/blob/896bf1af7782856cad6fcbfe4c93867252ee3548/sdks/go/pkg/beam/core/runtime/xlangx/registry.go#L289

And plumbed through to https://github.com/apache/beam/blob/d2fe48e8c142a3ffc16957b779f5e7bcc8a034ab/sdks/go/pkg/beam/core/runtime/xlangx/expand.go#L180 for the started service.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.