NPE in BigQuery client library in StreamingBuffer.toPb
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Sometimes I see the following NPE in my streaming pipeline:
```
Caused by: java.lang.NullPointerException com.google.cloud.bigquery.StandardTableDefinition$StreamingBuffer.toPb(StandardTableDefinition.java:106)
com.google.cloud.bigquery.StandardTableDefinition.toPb(StandardTableDefinition.java:230) com.google.cloud.bigquery.TableInfo.toPb(TableInfo.java:453)
com.google.cloud.bigquery.BigQueryImpl.update(BigQueryImpl.java:383) com.google.cloud.bigquery.Table.update(Table.java:212)
com.heysphere.nosegment.dofn.TableRowSchemaMutatorFn.processElement(TableRowSchemaMutatorFn.java:132)
```
It happens extremely rarely and I'm not sure what can cause it. To add a bit more context, the pipeline performs `Table.update` when it handles failed inserts from BigQueryIO and detects that the problem can be fixed by adding missing columns (sort of like https://github.com/ryanmcdowell/dataflow-dynamic-schema )
Somewhat reminds BEAM-6076
Imported from Jira [BEAM-6434](https://issues.apache.org/jira/browse/BEAM-6434). Original Jira may contain additional context.
Reported by: dkruchinin.
Contributor guide
Assessment
This issue has not been assessed yet.