apache / apache/beam

BigQueryIO.readTableRows() throws NullPointerException with TableReference and decorator

Open
#18,885 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When attaching a partition decorator to the table id, readTableRows() throws a NullPointerException. 
```

TableReference tableRef = BigQueryHelpers.parseTableSpec(options.getBqTable());
tableRef.setTableId(tableRef.getTableId()
+ "$" + options.getPartitionValue());

//Regardless of whether this should work with the table decorator,
//it
should not throw a NullPointerException
PCollection rowsByPartition = p.apply("Read live
data",
BigQueryIO.readTableRows().from(tableRef));

```

Exception:

Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at com.fearlesstg.CannotReadFromPartitionedTable.main(CannotReadFromPartitionedTable.java:26)
Caused by: java.lang.NullPointerException
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableSource.getEstimatedSizeBytes(BigQueryTableSource.java:108)
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:207)
at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)

 

Detailed example: 

[https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotReadFromPartitionedTable.java](https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotReadFromPartitionedTable.java)

 

Personally, I would also like this pattern to be supported as intended and not have to construct queries with IN / WHERE clauses to pull partitions.

Imported from Jira [BEAM-4369](https://issues.apache.org/jira/browse/BEAM-4369). Original Jira may contain additional context.
Reported by: pbrumblay.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.