apache / apache/incubator-xtable

Hudi 0.x writes INT64 in parquet for Avro LogicalTypes.LocalTimestampMillis, needs 1.x upgrade

Open
#672 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

When adding supporting for timestamp_ntz for delta and iceberg, found a bug where hudi 0.x is writing INT64 for LocalTimestampMillis and LocalTimestampMicros.

```
############ Column(timestamp_millis_nullable_field) ############
name: timestamp_millis_nullable_field
path: timestamp_millis_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false)
converted_type (legacy): TIMESTAMP_MILLIS
compression: UNCOMPRESSED (space_saved: 0%)

############ Column(timestamp_micros_nullable_field) ############
name: timestamp_micros_nullable_field
path: timestamp_micros_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=microseconds, is_from_converted_type=false, force_set_converted_type=false)
converted_type (legacy): TIMESTAMP_MICROS
compression: UNCOMPRESSED (space_saved: 0%)

############ Column(timestamp_local_millis_nullable_field) ############
name: timestamp_local_millis_nullable_field
path: timestamp_local_millis_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: UNCOMPRESSED (space_saved: 0%)

############ Column(timestamp_local_micros_nullable_field) ############
name: timestamp_local_micros_nullable_field
path: timestamp_local_micros_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: UNCOMPRESSED (space_saved: 0%)
```

https://github.com/apache/incubator-xtable/pull/647/files#r2023648267

Hudi 1.x has fixed the problem, adding this GH issue to revert the schema mapping back to timestamp_ntz in AvroSchemaConverter.
https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L152

### Are you willing to submit PR?

- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate AvroSchemaConverter and inspect the mapping for Avro LogicalTypes.LocalTimestampMillis and LocalTimestampMicros, then compare it with the Hudi 1.x behavior linked in the issue. Verify the mapping is restored to timestamp_ntz and that the resulting Parquet schema preserves the expected timestamp logical type.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.