apache / apache/incubator-xtable
Hudi 0.x writes INT64 in parquet for Avro LogicalTypes.LocalTimestampMillis, needs 1.x upgrade
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
When adding supporting for timestamp_ntz for delta and iceberg, found a bug where hudi 0.x is writing INT64 for LocalTimestampMillis and LocalTimestampMicros.
```
############ Column(timestamp_millis_nullable_field) ############
name: timestamp_millis_nullable_field
path: timestamp_millis_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false)
converted_type (legacy): TIMESTAMP_MILLIS
compression: UNCOMPRESSED (space_saved: 0%)
############ Column(timestamp_micros_nullable_field) ############
name: timestamp_micros_nullable_field
path: timestamp_micros_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=microseconds, is_from_converted_type=false, force_set_converted_type=false)
converted_type (legacy): TIMESTAMP_MICROS
compression: UNCOMPRESSED (space_saved: 0%)
############ Column(timestamp_local_millis_nullable_field) ############
name: timestamp_local_millis_nullable_field
path: timestamp_local_millis_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: UNCOMPRESSED (space_saved: 0%)
############ Column(timestamp_local_micros_nullable_field) ############
name: timestamp_local_micros_nullable_field
path: timestamp_local_micros_nullable_field
max_definition_level: 1
max_repetition_level: 0
physical_type: INT64
logical_type: None
converted_type (legacy): NONE
compression: UNCOMPRESSED (space_saved: 0%)
```
https://github.com/apache/incubator-xtable/pull/647/files#r2023648267
Hudi 1.x has fixed the problem, adding this GH issue to revert the schema mapping back to timestamp_ntz in AvroSchemaConverter.
https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L152
### Are you willing to submit PR?
- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate AvroSchemaConverter and inspect the mapping for Avro LogicalTypes.LocalTimestampMillis and LocalTimestampMicros, then compare it with the Hudi 1.x behavior linked in the issue. Verify the mapping is restored to timestamp_ntz and that the resulting Parquet schema preserves the expected timestamp logical type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100