apache / apache/beam

[Bug]: Calling a Java custom PTransform from Go using xlang fails on Dataflow Runner

Open
#23,189 2 comments 0 reactions 0 assignees View on GitHub
bug go P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

As mentioned in #22931 - I am testing xlang support for the Go Runtime. I first was doing it wrong, using the Go direct runner, but then I moved on to test with the Dataflow runner.

I noticed that on Dataflow Runner, I could call a custom [Java Ptransform](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/java/src/main/java/com/ronoaldo/SplitWordsFromJava.java) from [Python](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/pipeline.py):

![Screenshot_20220912_154742](https://user-images.githubusercontent.com/564649/189732415-986bc0b4-b6ed-406f-8874-e542095f24ba.png)

However,I could not call the same PTransform [from Go](https://github.com/ronoaldo/micro-beam/blob/main/05_xlang/go/pipeline.go). To submit the job, I started the Java expansion service as I did for the Python run, and called my Go pipeline. I see that the expansions service is called from Go, that the Go runtime stages the .jar files into Cloud Storage, but then the pipeline fails:

![image](https://user-images.githubusercontent.com/564649/189730927-c9181ea9-2def-4c32-8160-1290b1fb9767.png)

The only error message available is:

```
S03:e5/ProcessElementAndRestrictionWithSizing+External/FlatMapElements/FlatMap/ParMultiDo(Anonymous)+External/Filter/ParDo(Anonymous)/ParMultiDo(Anonymous)+stats.Count/stats.keyedCountFn+stats.Count/stats.SumPerKey/CombinePerKey/CoGBK+stats.Count/stats.SumPerKey/CombinePerKey/stats.sumIntFn/Partial+stats.Count/stats.SumPerKey/CombinePerKey/CoGBK/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:

go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,

go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,

go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.,

go-job-1-1662153285384063-09021416-tqko-harness-x8dv
Root cause: The worker lost contact with the service.
```

Here is a full [pipeline log](https://docs.google.com/spreadsheets/d/1OFKBdDiHsz2ZAu5K_DjP4g15Wr2tr3rroyLN3qmts8o/edit?usp=sharing) and [worker logs](https://docs.google.com/spreadsheets/d/1aN8ycCBvs3tJljZ2gN2J0rv8GypQxebIcfM7tygGEeU/edit?usp=sharing) downloaded from Logs explorer.

Is this an actual bug or a unsupported workflow? I am trying to follow the Multi language pipelines section of the [Beam Programming Model docs](https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines), regarding expose the Java PTransform (which I assume is correct since calling from Python works) and consuming it from Go, which I'm not sure if I missed any important steps.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.