apache / apache/beam

There is a NullPointerException when read null-value field in BigQuery table

Open
#19,340 0 comments 0 reactions 0 assignees View on GitHub
bug dsl P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I run query on a BigQuery table by Beam SQL.

Ex: 

```

BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery("SELECT * FROM X_bigquery_table"));

```

There is a NullPointerException when it reads null-value field in the BigQuery table as below:

```

Dec 22, 2018 11:05:21 AM org.apache.beam.sdk.io.FileBasedSource createReader
INFO: Matched 1 files
for pattern gs://xxx/tmp/BigQueryExtractTemp/a84545971aa94cf6b6717984e9d71642/000000000000.avro
java.lang.NullPointerException

at org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroString(AvroUtils.java:81)
at org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:104)

at org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:46)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:206)

at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:198)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:185)

at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:221)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:214)
at org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)

at org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)

at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
at
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
at org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)

at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)

at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)

at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

```

 

Imported from Jira [BEAM-6297](https://issues.apache.org/jira/browse/BEAM-6297). Original Jira may contain additional context.
Reported by: luat.

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.java at convertAvroString and trace its callers in BigQueryUtils.java. Reproduce the Beam SQL query against a BigQuery table containing a null field, then verify that conversion completes without the reported NullPointerException and preserves the null value.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.