apache / apache/hudi

CustomKeyGenerator can not be created with flink

Open
#16,621 1 comment 0 reactions 1 assignee Assigned to @danny0405 View on GitHub
from-jira priority:blocker type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

If we create a table in flink with custom key generator it fails with error message.
{code:java}
[ERROR] Could not execute SQL statement. Reason:
org.apache.hudi.exception.HoodieValidationException: Field ts:timestamp does not exist{code}
{code:java}
CREATE TABLE hudi_table(
    ts BIGINT,
    uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
    rider VARCHAR(20),
    driver VARCHAR(20),
    fare DOUBLE,
    city VARCHAR(20)
)
WITH (
  'connector' = 'hudi',
  'path' = 'file:///tmp/hudi_table',
  'table.type' = 'MERGE_ON_READ',
  'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
  'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
  'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
  'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.CustomAvroKeyGenerator',
  'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
); {code}
We need to support using custom keygen with flink engine.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8220
- Type: Bug
- Fix version(s):
- 1.1.0

---

## Comments

20/Sep/24 09:34;danny0405;Let's revisit this option value, it looks like a beaking change, you can not force user to always declare the very specific partition data type, there should be a default value, why not introduce a new option just for the custom ken gen partition data type?

{code:java}
'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp' {code};;;

---

23/Sep/24 04:46;ljain;Updated the exception message which is being seen.
[~danny0405] The problem here is that the user would not be able to specify a partition type even if they want to. The partition type is only required for custom keygen and not required for other keygens.;;;

---

27/Sep/24 04:21;danny0405;Moved it to 1.1.0 release because it looks like we never support the custom avro key gen for Flink before, so it is not a blocker. And I do think the special value for partition path field is quite hacky, I'm conservative about the modification.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.