apache / apache/beam

BigQueryIO result different for REPEATED field between DirectRunner and DataflowRunner

Open
#18,445 1 comment 0 reactions 0 assignees View on GitHub
bug dataflow direct gcp io java P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When running a query against BigQueryIO with a REPEATED RECORD field the behavior is different between DirectRunner and DataflowRunner. The field containing the repeated record has to be cast to access the records. Apparently the following implementations work for each runner but I would expect them to be the same as my pipeline otherwise only runs on one.

DirectRunner:

```

ArrayList orderLines = (ArrayList) c.element().get("RepeatedField");

```

DataflowRunner:

```

ImmutableList orderLines = (ImmutableList) c.element().get("RepeatedField");

```

For example when using the ImmutableList implementation on DirectRunner the following exception is thrown:

```

java.lang.ClassCastException: java.util.ArrayList cannot be cast to com.google.common.collect.ImmutableList

```

Imported from Jira [BEAM-2767](https://issues.apache.org/jira/browse/BEAM-2767). Original Jira may contain additional context.
Reported by: jroxtheworld.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the BigQueryIO handling of REPEATED RECORD fields in the DirectRunner and DataflowRunner paths, using the two result types and ClassCastException in the report as the reproduction. Compare the returned representations and add or update coverage so both runners expose the same usable result shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.