apache / apache/paimon

[Bug] Use hive cannot insert data from not partition table to a partition table.

Open
#4,550 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.9

### Compute Engine

Hive: CDH6.3.2

### Minimal reproduce step

```sql
SET hive.metastore.warehouse.dir=/user/hive/warehouse;

create table hive_test_np(
a int,
b string,
c string)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';

insert into hive_test_np values(1, 'aaa', 'nice');

create table hive_test(
a int,
b string)
PARTITIONED BY ( c string)
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler';

insert into hive_test partition(c) select * from hive_test_np;
```

### What doesn't meet your expectations?

```
2024-11-19 21:45:16,486 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"a":1,"b":"aaa","c":"nice"}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"a":1,"b":"aaa","c":"nice"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:494)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
... 8 more
Caused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.paimon.hive.mapred.PaimonRecordWriter.write(PaimonRecordWriter.java:69)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:769)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:146)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:484)
... 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.paimon.data.GenericRow.isNullAt(GenericRow.java:131)
at Projection$0.apply(Unknown Source)
at org.apache.paimon.table.sink.RowPartitionKeyExtractor.partition(RowPartitionKeyExtractor.java:44)
at org.apache.paimon.table.sink.RowKeyExtractor.partition(RowKeyExtractor.java:57)
at org.apache.paimon.table.sink.TableWriteImpl.toSinkRecord(TableWriteImpl.java:204)
at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:174)
at org.apache.paimon.table.sink.TableWriteImpl.write(TableWriteImpl.java:147)
at org.apache.paimon.hive.mapred.PaimonRecordWriter.write(PaimonRecordWriter.java:67)
... 16 more

```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Hive CDH6.3.2 case from the SQL shown, then inspect PaimonRecordWriter.java and the RowPartitionKeyExtractor/RowKeyExtractor stack frames. Trace how the non-partitioned source row is projected for the partitioned target. Done means the insert completes without ArrayIndexOutOfBoundsException and writes the row to the expected partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.