apache / apache/hudi

[SUPPORT] Flink Hudi Job Configuration and Parameter Conflict Issues

Open
#12,024 12 comments 0 reactions 0 assignees View on GitHub
area:config
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

1. **Configuration Conflict in Flink Hudi Job**: When modifying the configuration of an existing Flink Hudi Job, if there is a conflict with the Table Config (hoodie.properties), the job does not throw an error. Instead, it silently overrides the user configuration with the previous Table Config settings.
2. **Parameter Conflict Handling**: When certain parameters conflict, the job does not check for these conflicts at startup. Errors are only thrown during runtime, which delays problem detection.
3. **Flink SQL Keywords Conflict**: When the Flink SQL keywords `PRIMARY KEY` and `PARTITIONED BY` conflict with user configurations `hoodie.datasource.write.recordkey.field` and `hoodie.datasource.write.partitionpath.field`, the job does not throw an error. Instead, it prioritizes the Flink SQL keywords over the Hoodie configurations.

**To Reproduce**

Steps to reproduce the behavior:

1. **Configuration Conflict in Flink Hudi Job**
1. Start a Flink Hudi Job and create `hoodie.properties`.
2. Modify the Flink Hudi Table configuration and restart the job.
3. Observe that the modifications do not take effect and the job starts normally without errors.

2. **Parameter Conflict Handling**
1. Configure the record key field with two fields: `hoodie.datasource.write.recordkey.field = 'uuid,name'`.
2. Configure a mismatched key generator: `hoodie.datasource.write.keygenerator.class='org.apache.hudi.keygen.SimpleAvroKeyGenerator'`.
3. The job will throw an error during runtime, indicating that the Avro Record cannot find the fields 'uuid,name'.

3. **Flink SQL Keywords Conflict**
1. Create a Flink table:
```sql
CREATE TABLE t_test (
`uuid` VARCHAR(20),
`name` VARCHAR(10),
`age` INT,
`ts` TIMESTAMP(3),
`partition` VARCHAR(10),
PRIMARY KEY (uuid, name) NOT ENFORCED
)
PARTITIONED BY (`partition`)
WITH (
'hoodie.datasource.write.recordkey.field' = 'age',
'hoodie.datasource.write.partitionpath.field' = 'name'
);
```
2. Observe that the job does not throw an error and internally prioritizes `PRIMARY KEY` and `PARTITIONED BY` over the Hoodie Config settings.

**Expected behavior**

Discussion item: I'd like to ask whether we should strictly check for configuration conflicts. Should we directly report an error in case of a conflict, rather than internally modifying user parameters?

I prefer directly reporting an error. I have a reason: if we don't report an error, users might mistakenly believe their configuration is valid, which could lead to confusion.

**Environment Description**

* Hudi version : 0.15.0

* Spark version : none

* Hive version : none

* Hadoop version : 3.3.5

* Storage (HDFS/S3/GCS..) : HDFS

* Running on Docker? (yes/no) : no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Flink Hudi table-configuration path and the hoodie.properties behavior described in the reproduction; compare it with the CREATE TABLE ... WITH example. Reproduce the startup and runtime conflict cases, then confirm the intended completion criterion, since the issue remains a discussion about reporting conflicts instead of silently overriding parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.