apache / apache/hudi

[SUPPORT] Deltastreamer throws exception when ingesting INT96 timestamps

Open
#9,151 3 comments 0 reactions 0 assignees View on GitHub
area:schema engine:spark
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Deltastreamer throws an exception when trying to ingest a ParquetDFSSource with INT96 timestamps.

After adding `"spark.sql.legacy.parquet.int96RebaseModeInRead": "CORRECTED"` and `"spark.sql.legacy.avro.datetimeRebaseModeInWrite": "CORRECTED"` as configs the ingest succeeds only if pointed at a source.dfs.root with a single parquet file, attempting to ingest multiple parquet files throws the error mentioned in the stacktrace.

**To Reproduce**

Steps to reproduce the behavior:

1. Run Deltastreamer on a ParquetDFSSource with a single file that have INT96 timestamps with no legacy datetime rebase configs
2. Observe error
3. Rerun with legacy configs enabled, ingest should succeed.
4. Run Deltastreamer again with the legacy configs, but with multiple parquet files with INT96 timestamps.

**Expected behavior**

Enabling the two legacy configs should allow for the ingest of all parquet files at the source.dfs.root that contain INT96 timestamps.

**Environment Description**

Hudi version : 13

Spark version : 3.1

Hive version : N/A

Hadoop version : N/A

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : yes

**Additional context**

Issue mentioned previously [here](https://github.com/apache/hudi/issues/6278)

Comment mentioning issue ingesting multiple parquet files with legacy datetime rebase configs enabled [here](https://github.com/apache/hudi/issues/6278#issuecomment-1356756712)

Have tried setting the configs to "LEGACY" instead of "CORRECTED", but the same error is thrown. Have also tried `spark.sql.parquet.int96RebaseModeInRead` and `spark.sql.avro.datetimeRebaseModeInWrite` instead of sql.legacy, but the original error is thrown as well.

**Stacktrace**

```org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet INT96 files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.int96RebaseModeInRead to 'CORRECTED' to read the datetime values as it is.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Deltastreamer ParquetDFSSource reproduction described in the issue, using multiple Parquet files containing INT96 timestamps and the two listed Spark rebase configurations. Compare the single-file and multi-file runs and inspect the resulting stacktrace. Done means all matching files under source.dfs.root ingest successfully with the configurations enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.