CustomKeyGenerator can not be created with flink
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
If we create a table in flink with custom key generator it fails with error message.
{code:java}
[ERROR] Could not execute SQL statement. Reason:
org.apache.hudi.exception.HoodieValidationException: Field ts:timestamp does not exist{code}
{code:java}
CREATE TABLE hudi_table(
ts BIGINT,
uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
rider VARCHAR(20),
driver VARCHAR(20),
fare DOUBLE,
city VARCHAR(20)
)
WITH (
'connector' = 'hudi',
'path' = 'file:///tmp/hudi_table',
'table.type' = 'MERGE_ON_READ',
'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.CustomAvroKeyGenerator',
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
); {code}
We need to support using custom keygen with flink engine.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8220
- Type: Bug
- Fix version(s):
- 1.1.0
---
## Comments
20/Sep/24 09:34;danny0405;Let's revisit this option value, it looks like a beaking change, you can not force user to always declare the very specific partition data type, there should be a default value, why not introduce a new option just for the custom ken gen partition data type?
{code:java}
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp' {code};;;
---
23/Sep/24 04:46;ljain;Updated the exception message which is being seen.
[~danny0405] The problem here is that the user would not be able to specify a partition type even if they want to. The partition type is only required for custom keygen and not required for other keygens.;;;
---
27/Sep/24 04:21;danny0405;Moved it to 1.1.0 release because it looks like we never support the custom avro key gen for Flink before, so it is not a blocker. And I do think the special value for partition path field is quite hacky, I'm conservative about the modification.;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.