apache / apache/beam

[Bug]: Cross-Language pipelines doesn't work with BigQuery IO and Go SDK on Dataflow Runner

Open
#22,172 5 comments 0 reactions 0 assignees View on GitHub
bug cross-language go io P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

I am using Go SDK and `github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/bigqueryio` package.

When i am running a pipeline which writes to BigQuery using `xlang` package, it always fails with error

```
Error message from worker: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn
```

That is also reproducable with this wordcount example:
https://github.com/apache/beam/blob/master/sdks/go/examples/xlang/bigquery/wordcount.go

```
Error message from worker: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead$4
```

I might be doing something wrong, but that looks to me as lack of Java SDK on runner or some kind of other runner issues.

### Issue Priority

Priority: 2

### Issue Component

Component: cross-language

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.