apache / apache/hudi

CustomKeyGenerator with TIMESTAMP skipping the microseconds part

Open
#17,247 1 comment 0 reactions 1 assignee Claimed by @danny0405 View on GitHub
area:writer from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

CustomKeyGenerator is not able to parse the microseconds part. Below is the reproducible code. In output its giving - "2023-03-04 14:44:42.046000"
```
fake = Faker()
data = [{"ID": fake.uuid4(), "EventTime": "2023-03-04 14:44:42.046661",
"FullName": fake.name(), "Address": fake.address(),
"CompanyName": fake.company(), "JobTitle": fake.job(),
"EmailAddress": fake.email(), "PhoneNumber": fake.phone_number(),
"RandomText": fake.sentence(), "City": fake.city(),
"State": fake.state(), "Country": fake.country()} for _ in range(5)]
pandas_df = pd.DataFrame(data)

hoodie_properties = {
'hoodie.table.name': "pj_poc",
'hoodie.datasource.write.recordkey.field': 'ID',
'hoodie.datasource.write.partitionpath.field': 'State:SIMPLE,Country:SIMPLE,EventTime:TIMESTAMP',
'hoodie.datasource.write.table.name': "pj_poc",
'hoodie.datasource.write.precombine.field': 'EventTime',
'hoodie.datasource.write.hive_style_partitioning':'true',
'hoodie.datasource.write.keygenerator.class':'org.apache.hudi.keygen.CustomKeyGenerator',
'hoodie.keygen.timebased.input.dateformat':'yyyy-MM-dd HH:mm:ss.SSSSSS',
'hoodie.keygen.timebased.output.dateformat':'yyyy-MM-dd HH:mm:ss.SSSSSS',
'hoodie.keygen.timebased.timestamp.type' : 'DATE_STRING',
'hoodie.keygen.timebased.timestamp.scalar.time.unit': 'MICROSECONDS',
'hoodie.parquet.outputtimestamptype': 'TIMESTAMP_MICROS',
}

spark.sparkContext.setLogLevel("WARN")

df = spark.createDataFrame(pandas_df)
df.write.format("hudi").options(**hoodie_properties).mode("overwrite").save(PATH)
spark.read.options(**hoodie_properties).format("hudi").load(PATH).select("_hoodie_partition_path", "EventTime").show(10, False)
```

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7306
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.