[Bug] Use hive cannot insert data from not partition table to a partition table.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.9
### Compute Engine
Hive: CDH6.3.2
### Minimal reproduce step
```sql
SET hive.metastore.warehouse.dir=/user/hive/warehouse;
create table hive_test_np(
a int,
b string,
c string)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';
insert into hive_test_np values(1, 'aaa', 'nice');
create table hive_test(
a int,
b string)
PARTITIONED BY ( c string)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';
insert into hive_test partition(c) select * from hive_test_np;
```
### What doesn't meet your expectations?
```
2024-11-19 21:45:16,486 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"a":1,"b":"aaa","c":"nice"}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"a":1,"b":"aaa","c":"nice"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:494)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.paimon.hive.mapred.PaimonRecordWriter.write(PaimonRecordWriter.java:69)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:769)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:146)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:484)
... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.paimon.data.GenericRow.isNullAt(GenericRow.java:131)
at Projection$0.apply(Unknown Source)
at org.apache.paimon.table.sink.RowPartitionKeyExtractor.partition(RowPartitionKeyExtractor.java:44)
at org.apache.paimon.table.sink.RowKeyExtractor.partition(RowKeyExtractor.java:57)
at org.apache.paimon.table.sink.TableWriteImpl.toSinkRecord(TableWriteImpl.java:204)
at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:174)
at org.apache.paimon.table.sink.TableWriteImpl.write(TableWriteImpl.java:147)
at org.apache.paimon.hive.mapred.PaimonRecordWriter.write(PaimonRecordWriter.java:67)
... 16 more
```
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Hive CDH6.3.2 case from the SQL shown, then inspect PaimonRecordWriter.java and the RowPartitionKeyExtractor/RowKeyExtractor stack frames. Trace how the non-partitioned source row is projected for the partitioned target. Done means the insert completes without ArrayIndexOutOfBoundsException and writes the row to the expected partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100