[SUPPORT] Hudi V0.9 truncating second precision for timestamp columns

Hi Team,

Using the configs below, I see Hudi is truncating the second precisions while ingesting the data. We are currently on 0.9V and I have observed this issue with this version, but it worked with 0.11V.

Do I need to add any other configurations to make it work with 0.9V without migrating to 0.11V? Any help on how to avoid this issue would be greatly appreciated.

Configs

db_name = tst_db
tableName =tst_tb
pk = ‘id’
de_dup = ‘last_updated’
commonConfig = {
“hoodie.datasource.hive_sync.database”: db_name,
‘hoodie.table.name’: tableName,
‘hoodie.datasource.hive_sync.support_timestamp’: ‘true’,
‘hoodie.datasource.write.recordkey.field’: pk,
‘hoodie.datasource.write.precombine.field’: de_dup,
‘hoodie.datasource.hive_sync.enable’: ‘true’,
‘hoodie.datasource.hive_sync.table’: tableName
}
nonPartitionConfig = {
‘hoodie.datasource.hive_sync.partition_extractor_class’:
‘org.apache.hudi.hive.NonPartitionedExtractor’,
‘hoodie.datasource.write.keygenerator.class’:
‘org.apache.hudi.keygen.NonpartitionedKeyGenerator’
}
config = {
‘hoodie.bulkinsert.shuffle.parallelism’: 10,
‘hoodie.datasource.write.operation’: ‘bulk_insert’
}
S3Location = ‘s3://<>/hudi/tst_tb’
combinedConf = {**commonConfig, **nonPartitionConfig, **config}
df.write.format(‘org.apache.hudi’).options(
**combinedConf).mode(‘overwrite’).save(S3Location)

Environment Description

EMR: emr-6.5.0
Hudi version : 0.9
Spark version : Spark 3.1.2
Hive version : Hive 3.1.2
Hadoop version :Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no

Source Data

+----------+--------------------------+--------------------------+
|id |creation_date |last_updated |
+----------+--------------------------+--------------------------+
|7cb15b859e|2021-11-07 08:48:25.000232|2021-11-08 08:50:35.000359|
|60ab5da73a|2022-07-02 19:48:27.000891|2022-07-03 20:05:19.000364|
|abb663a826|2015-07-12 15:35:14 |2015-08-01 15:38:07 |
|c92aaeedc1|2021-05-10 16:47:10.000455|2021-05-30 16:49:29.00063 |
+----------+--------------------------+--------------------------+

Source Schema

root
|-- id: string (nullable = true)
|-- creation_date: timestamp (nullable = true)
|-- last_updated: timestamp (nullable = true)

Hudi 0.9V Output

+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+----------+-------------------+-------------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |creation_date |last_updated |
+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+----------+-------------------+-------------------+
|20220728035114 |20220728035114_3_2 |c92aaeedc1 | |1736fb90-f6b2-4282-9c77-da2ace4bf0bd-0_3-10-80_20220728035114.parquet|c92aaeedc1|2021-05-10 16:47:10|2021-05-30 16:49:29|
|20220728035114 |20220728035114_1_3 |7cb15b859e | |d650a502-386e-47b9-81f3-e72cf64b0c0e-0_1-10-78_20220728035114.parquet|7cb15b859e|2021-11-07 08:48:25|2021-11-08 08:50:35|
|20220728035114 |20220728035114_2_1 |abb663a826 | |941ca621-111e-47d9-8ca1-bdc943490371-0_2-10-79_20220728035114.parquet|abb663a826|2015-07-12 15:35:14|2015-08-01 15:38:07|
|20220728035114 |20220728035114_0_1 |60ab5da73a | |2d2fb872-7775-4b2d-bd28-93c289ae12c8-0_0-8-77_20220728035114.parquet |60ab5da73a|2022-07-02 19:48:27|2022-07-03 20:05:19|
+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+----------+-------------------+-------------------+

Hudi 0.11V Output

+-------------------+---------------------+------------------+----------------------+-------------------------------------------------------------------------+----------+--------------------------+--------------------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |creation_date |last_updated |
+-------------------+---------------------+------------------+----------------------+-------------------------------------------------------------------------+----------+--------------------------+--------------------------+
|20220728035802662 |20220728035802662_0_1|1340225 | |38263eea-aa5d-4adf-b7f1-f11ebd2f9142-0_0-2522-0_20220728035802662.parquet|1340225 |2017-01-24 00:02:10 |2022-02-25 07:03:54.000853|
|20220728035802662 |20220728035802662_0_2|53773de3-9 | |38263eea-aa5d-4adf-b7f1-f11ebd2f9142-0_0-2522-0_20220728035802662.parquet|53773de3-9|2022-02-25 07:21:06.000037|2022-02-25 08:35:57.000877|
|20220728035802662 |20220728035802662_0_3|722b232f-e | |38263eea-aa5d-4adf-b7f1-f11ebd2f9142-0_0-2522-0_20220728035802662.parquet|722b232f-e|2022-02-22 06:02:32.000481|2022-02-25 08:54:05.00042 |
+-------------------+---------------------+------------------+----------------------+-------------------------------------------------------------------------+----------+--------------------------+--------------------------+