apache / apache/paimon

[Bug] Can't extract bucket from row in dynamic bucket when using hive(mr or tez) insert into paimon primary table with dynamic-bucket

Open
#5,123 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.0-20241111

### Compute Engine

hive3.1.2

### Minimal reproduce step

1)sparksql
CREATE TABLE spark_catalog.tmp.hive2paimon_withpk (
receive_time BIGINT NOT NULL,
source_ip STRING NOT NULL,
source_port INT,
sdk_version STRING)
USING paimon
TBLPROPERTIES (
'bucket' = '-1',
'dynamic-bucket.initial-buckets' = '1',
'dynamic-bucket.target-row-num' = '1000',
'file.format' = 'orc',
'primary-key' = 'receive_time,source_ip')

2)hive
insert into tmp.hive2paimon_withpk select 1740016451,'192.168.100.200',34738,'2.9.9';

### What doesn't meet your expectations?

org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:234) ... 7 more Caused by: java.lang.IllegalArgumentException: Can't extract bucket from row in dynamic bucket mode, you should use 'TableWrite.write(InternalRow row, int bucket)' method. at org.apache.paimon.table.sink.DynamicBucketRowKeyExtractor.bucket(DynamicBucketRowKeyExtractor.java:44) at org.apache.paimon.table.sink.TableWriteImpl.toSinkRecord(TableWriteImpl.java:205) at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:174) at org.apache.paimon.table.sink.TableWriteImpl.write(TableWriteImpl.java:147) at org.apache.paimon.hive.mapred.PaimonRecordWriter.write(PaimonRecordWriter.java:67) ... 17 more

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Hive insert using the provided Paimon table definition and dynamic-bucket properties. Start with PaimonRecordWriter.java, then trace TableWriteImpl.java and DynamicBucketRowKeyExtractor.java to determine why the row lacks a bucket in this path. Done means the MR or Tez insert completes without the IllegalArgumentException and writes the row successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.