BigQueryIO.readTableRows() throws NullPointerException with TableReference and decorator
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When attaching a partition decorator to the table id, readTableRows() throws a NullPointerException.
```
TableReference tableRef = BigQueryHelpers.parseTableSpec(options.getBqTable());
tableRef.setTableId(tableRef.getTableId()
+ "$" + options.getPartitionValue());
//Regardless of whether this should work with the table decorator,
//it
should not throw a NullPointerException
PCollection rowsByPartition = p.apply("Read live
data",
BigQueryIO.readTableRows().from(tableRef));
```
Exception:
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at com.fearlesstg.CannotReadFromPartitionedTable.main(CannotReadFromPartitionedTable.java:26)
Caused by: java.lang.NullPointerException
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableSource.getEstimatedSizeBytes(BigQueryTableSource.java:108)
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:207)
at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)
Detailed example:
[https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotReadFromPartitionedTable.java](https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotReadFromPartitionedTable.java)
Personally, I would also like this pattern to be supported as intended and not have to construct queries with IN / WHERE clauses to pull partitions.
Imported from Jira [BEAM-4369](https://issues.apache.org/jira/browse/BEAM-4369). Original Jira may contain additional context.
Reported by: pbrumblay.
Contributor guide
Assessment
This issue has not been assessed yet.