[SUPPORT]NullPointerException in query x_ro table by hive
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
After create a mor table using spark and insert some data into the table, the NPE occur when query the x_ro table by hive
**To Reproduce**
Steps to reproduce the behavior:
1. create mor table in spark and insert data to table
create table default.hudi_mor_tbl_noauth (id int, name string, price double, ts bigint) using hudi tblproperties ( type = 'mor', primaryKey = 'id', preCombineField = 'ts') location '/tmp/hudi-mor/hudi_mor_tbl_noauth';
insert into default.hudi_mor_tbl_noauth values(1, "name", 100.1, 20230103);
2.query in hive
hive -hiveconf hive.root.logger=DEBUG,console -e "select * from hudi_mor_tbl_noauth_ro" >czm.out 2>czm.err
**Environment Description**
* Hudi version : 0.11.0
* Spark version :3.2
* Hive version :3.1.1
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception nulljava.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchOperator.getPartitionKeyOI(FetchOperator.java:243)
at org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:667)
at org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:187)
at org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:151)
at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:86)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:689)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Spark 3.2 and Hive 3.1.1 setup using the provided MOR table and query, then start with the Hive stack-trace entry point FetchOperator.getPartitionKeyOI in FetchOperator.java. Trace the metadata passed from the Hudi MOR table into Hive and identify the cause of the null value; done means selecting from hudi_mor_tbl_noauth_ro completes without the NullPointerException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100