apache / apache/hudi

[SUPPORT]NullPointerException in query x_ro table by hive

Open
#7,635 2 comments 1 reaction 0 assignees View on GitHub
area:query-engine engine:hive priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**
After create a mor table using spark and insert some data into the table, the NPE occur when query the x_ro table by hive

**To Reproduce**

Steps to reproduce the behavior:

1. create mor table in spark and insert data to table
create table default.hudi_mor_tbl_noauth (id int, name string, price double, ts bigint) using hudi tblproperties ( type = 'mor', primaryKey = 'id', preCombineField = 'ts') location '/tmp/hudi-mor/hudi_mor_tbl_noauth';

insert into default.hudi_mor_tbl_noauth values(1, "name", 100.1, 20230103);
2.query in hive
hive -hiveconf hive.root.logger=DEBUG,console -e "select * from hudi_mor_tbl_noauth_ro" >czm.out 2>czm.err

**Environment Description**

* Hudi version : 0.11.0

* Spark version :3.2

* Hive version :3.1.1

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception nulljava.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchOperator.getPartitionKeyOI(FetchOperator.java:243)
at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:667)
at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:187)
at org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:151)
at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:86)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:689)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Spark 3.2 and Hive 3.1.1 setup using the provided MOR table and query, then start with the Hive stack-trace entry point FetchOperator.getPartitionKeyOI in FetchOperator.java. Trace the metadata passed from the Hudi MOR table into Hive and identify the cause of the null value; done means selecting from hudi_mor_tbl_noauth_ro completes without the NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.