apache / apache/hudi

Fail to bootstrap/upsert a table which contains timestamp column

Open
#14,777 2 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:build engine:spark from-jira priority:blocker status:pr-available type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

current when hudi bootstrap a parquet file, or upsert into a parquet file which contains timestmap column, it will fail because these issues:

1) At bootstrap operation, if the origin parquet file was written by a spark application, then spark will default save timestamp as int96(see spark.sql.parquet.int96AsTimestamp), then bootstrap will fail, it’s because of Hudi can not read Int96 type now.(this issue can be solve by upgrade parquet to 1.12.0, and set parquet.avro.readInt96AsFixed=true, please check [https://github|https://github/] <[https://github/]>.com/apache/parquet-mr/pull/831/files) 

2) after bootstrap, doing upsert will fail because we use hoodie schema to read origin parquet file. The schema is not match because hoodie schema  treat timestamp as long and at origin file it’s Int96 

3) after bootstrap, and partial update for a parquet file will fail, because we copy the old record and save by hoodie schema( we miss a convertFixedToLong operation like spark does)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1779
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1265
- Fix version(s):
- 1.1.0
- Attachment(s):
- 08/Apr/21 13:32;lrz;unsupportInt96.png;https://issues.apache.org/jira/secure/attachment/13023562/unsupportInt96.png
- 08/Apr/21 13:32;lrz;upsertFail.png;https://issues.apache.org/jira/secure/attachment/13023563/upsertFail.png
- 08/Apr/21 13:32;lrz;upsertFail2.png;https://issues.apache.org/jira/secure/attachment/13023564/upsertFail2.png

---

## Comments

08/Aug/21 20:17;githubbot;hudi-bot edited a comment on pull request #2790:
URL: https://github.com/apache/hudi/pull/2790#issuecomment-869766484


## CI report:

* 41aec7191e0345c1d8a4efb805ea04e1510bd480 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=502)


Bot commands
The @flinkbot bot supports the following commands:

- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org
;;;

---

09/Aug/21 04:22;githubbot;hudi-bot edited a comment on pull request #2790:
URL: https://github.com/apache/hudi/pull/2790#issuecomment-869766484


## CI report:

* 41aec7191e0345c1d8a4efb805ea04e1510bd480 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=502)


Bot commands
@hudi-bot supports the following commands:

- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org
;;;

---

13/Dec/21 14:26;shivnarayan;[~alexey.kudinkin] :  Related to parquet upgrade.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.