apache / apache/hudi

[SUPPORT]problem when inserting data to a non-partitioned table created by flink sql via spark sql cli

Open
#12,013 10 comments 0 reactions 0 assignees View on GitHub
area:writer engine:flink priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am using Hudi 0.15.0 and Flink 1.17.1, following are the steps to reproduce the problem:

From the flink-sql cli: do the following sql statements

```
CREATE CATALOG hudi_catalog WITH (
'type' = 'hudi',
'mode' = 'hms',
'default-database' = 'default',
'hive.conf.dir' = '/home/hadoop/software/hive-3.1.3/conf',
'table.external' = 'true'
);

create database if not exists hudi_catalog.`default`;

use hudi_catalog.`default`;

CREATE TABLE test_hudi_flink_mor_2 (
a int PRIMARY KEY NOT ENFORCED,
b int,
c int

)
WITH (
'connector' = 'hudi',
'path' = '/tmp/test_hudi_flink_mor_2',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator', --- Using ComplexAvroKeyGenerator
'hoodie.datasource.write.recordkey.field' = 'a',
'write.precombine.key'='b',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
'hive_sync.conf.dir'='/home/hadoop/software/hive-3.1.3/conf'
);

insert into test_hudi_flink_mor_2 values (1,1,1),(2,2,2);

```

So far so good, then I run `insert into test_hudi_flink_mor_2 select 3,3, 3` on the spark-sql cli, an error occurred, key exception message is as follows:

```

org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
KeyGenerator: org.apache.hudi.keygen.ComplexAvroKeyGenerator org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:229)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:232)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)

```

When I run `show create table test_hudi_flink_mor_2` on spark-sql cli, it shows that keygenerator is using ComplexAvroKeyGenerator,but when i look at the `/tmp/test_hudi_flink_mor_2/.hoodie/hoodie.properties`,
`hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedAvroKeyGenerator`, per the exception, it looks to me that there is a bug here...

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the Flink SQL and Spark SQL statements in the report, then inspect HoodieWriterUtils.scala and HoodieSparkSqlWriter.scala around the reported validation calls. Compare the catalog definition, SHOW CREATE TABLE output, and .hoodie/hoodie.properties for the key-generator values; done means the cross-engine insert no longer reports a configuration conflict.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.