[SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am using Hudi 0.15.0 and Flink 1.17.1, following are the steps to reproduce the problem:
From the flink-sql cli: do the following sql statements
```
CREATE CATALOG hudi_catalog WITH (
'type' = 'hudi',
'mode' = 'hms',
'default-database' = 'default',
'hive.conf.dir' = '/home/hadoop/software/hive-3.1.3/conf',
'table.external' = 'true'
);
create database if not exists hudi_catalog.`default`;
use hudi_catalog.`default`;
CREATE TABLE test_hudi_flink_mor_2 (
a int PRIMARY KEY NOT ENFORCED,
b int,
c int
)
WITH (
'connector' = 'hudi',
'path' = '/tmp/test_hudi_flink_mor_2',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator', --- Using ComplexAvroKeyGenerator
'hoodie.datasource.write.recordkey.field' = 'a',
'write.precombine.key'='b',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
'hive_sync.conf.dir'='/home/hadoop/software/hive-3.1.3/conf'
);
insert into test_hudi_flink_mor_2 values (1,1,1),(2,2,2);
```
So far so good, then I run `insert into test_hudi_flink_mor_2 select 3,3, 3` on the spark-sql cli, an error occurred, key exception message is as follows:
```
org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
KeyGenerator: org.apache.hudi.keygen.ComplexAvroKeyGenerator org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:229)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:232)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
```
When I run `show create table test_hudi_flink_mor_2` on spark-sql cli, it shows that keygenerator is using ComplexAvroKeyGenerator,but when i look at the `/tmp/test_hudi_flink_mor_2/.hoodie/hoodie.properties`,
`hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator`, per the exception, it looks to me that there is a bug here...
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue with the Flink SQL and Spark SQL statements in the report, then inspect HoodieWriterUtils.scala and HoodieSparkSqlWriter.scala around the reported validation calls. Compare the catalog definition, SHOW CREATE TABLE output, and .hoodie/hoodie.properties for the key-generator values; done means the cross-engine insert no longer reports a configuration conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100