[SUPPORT] Deltastreamer throws exception when ingesting INT96 timestamps
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Deltastreamer throws an exception when trying to ingest a ParquetDFSSource with INT96 timestamps.
After adding `"spark.sql.legacy.parquet.int96RebaseModeInRead": "CORRECTED"` and `"spark.sql.legacy.avro.datetimeRebaseModeInWrite": "CORRECTED"` as configs the ingest succeeds only if pointed at a source.dfs.root with a single parquet file, attempting to ingest multiple parquet files throws the error mentioned in the stacktrace.
**To Reproduce**
Steps to reproduce the behavior:
1. Run Deltastreamer on a ParquetDFSSource with a single file that have INT96 timestamps with no legacy datetime rebase configs
2. Observe error
3. Rerun with legacy configs enabled, ingest should succeed.
4. Run Deltastreamer again with the legacy configs, but with multiple parquet files with INT96 timestamps.
**Expected behavior**
Enabling the two legacy configs should allow for the ingest of all parquet files at the source.dfs.root that contain INT96 timestamps.
**Environment Description**
Hudi version : 13
Spark version : 3.1
Hive version : N/A
Hadoop version : N/A
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : yes
**Additional context**
Issue mentioned previously [here](https://github.com/apache/hudi/issues/6278)
Comment mentioning issue ingesting multiple parquet files with legacy datetime rebase configs enabled [here](https://github.com/apache/hudi/issues/6278#issuecomment-1356756712)
Have tried setting the configs to "LEGACY" instead of "CORRECTED", but the same error is thrown. Have also tried `spark.sql.parquet.int96RebaseModeInRead` and `spark.sql.avro.datetimeRebaseModeInWrite` instead of sql.legacy, but the original error is thrown as well.
**Stacktrace**
```org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet INT96 files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.int96RebaseModeInRead to 'CORRECTED' to read the datetime values as it is.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Deltastreamer ParquetDFSSource reproduction described in the issue, using multiple Parquet files containing INT96 timestamps and the two listed Spark rebase configurations. Compare the single-file and multi-file runs and inspect the resulting stacktrace. Done means all matching files under source.dfs.root ingest successfully with the configurations enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100