[SUPPORT] Flink Hudi Job Configuration and Parameter Conflict Issues
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
1. **Configuration Conflict in Flink Hudi Job**: When modifying the configuration of an existing Flink Hudi Job, if there is a conflict with the Table Config (hoodie.properties), the job does not throw an error. Instead, it silently overrides the user configuration with the previous Table Config settings.
2. **Parameter Conflict Handling**: When certain parameters conflict, the job does not check for these conflicts at startup. Errors are only thrown during runtime, which delays problem detection.
3. **Flink SQL Keywords Conflict**: When the Flink SQL keywords `PRIMARY KEY` and `PARTITIONED BY` conflict with user configurations `hoodie.datasource.write.recordkey.field` and `hoodie.datasource.write.partitionpath.field`, the job does not throw an error. Instead, it prioritizes the Flink SQL keywords over the Hoodie configurations.
**To Reproduce**
Steps to reproduce the behavior:
1. **Configuration Conflict in Flink Hudi Job**
1. Start a Flink Hudi Job and create `hoodie.properties`.
2. Modify the Flink Hudi Table configuration and restart the job.
3. Observe that the modifications do not take effect and the job starts normally without errors.
2. **Parameter Conflict Handling**
1. Configure the record key field with two fields: `hoodie.datasource.write.recordkey.field = 'uuid,name'`.
2. Configure a mismatched key generator: `hoodie.datasource.write.keygenerator.class='org.apache.hudi.keygen.SimpleAvroKeyGenerator'`.
3. The job will throw an error during runtime, indicating that the Avro Record cannot find the fields 'uuid,name'.
3. **Flink SQL Keywords Conflict**
1. Create a Flink table:
```sql
CREATE TABLE t_test (
`uuid` VARCHAR(20),
`name` VARCHAR(10),
`age` INT,
`ts` TIMESTAMP(3),
`partition` VARCHAR(10),
PRIMARY KEY (uuid, name) NOT ENFORCED
)
PARTITIONED BY (`partition`)
WITH (
'hoodie.datasource.write.recordkey.field' = 'age',
'hoodie.datasource.write.partitionpath.field' = 'name'
);
```
2. Observe that the job does not throw an error and internally prioritizes `PRIMARY KEY` and `PARTITIONED BY` over the Hoodie Config settings.
**Expected behavior**
Discussion item: I'd like to ask whether we should strictly check for configuration conflicts. Should we directly report an error in case of a conflict, rather than internally modifying user parameters?
I prefer directly reporting an error. I have a reason: if we don't report an error, users might mistakenly believe their configuration is valid, which could lead to confusion.
**Environment Description**
* Hudi version : 0.15.0
* Spark version : none
* Hive version : none
* Hadoop version : 3.3.5
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Flink Hudi table-configuration path and the hoodie.properties behavior described in the reproduction; compare it with the CREATE TABLE ... WITH example. Reproduce the startup and runtime conflict cases, then confirm the intended completion criterion, since the issue remains a discussion about reporting conflicts instead of silently overriding parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100