apache / apache/hudi

TIMESTAMP_MICROS handling

Open
#17,222 1 comment 0 reactions 1 assignee Claimed by @danny0405 View on GitHub
from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi Guys!
 
I am not able to use timestamp micro columns save with HUDI. 
I would like to save it keeping microsec granularity, but it only keeps milisec.
 
I have set this:
--conf spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS \
and also this in the hoodie:
"hoodie.parquet.outputtimestamptype": "TIMESTAMP_MICROS",
but when I read it back (with pyspark, load api), it's only millisecond precision and unfortunately, I need the microsec in some case, because with this I run into a Schrödinger's cat situation  !https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f604.png!
So an entity has more than one states in the same time !https://a.slack-edge.com/production-standard-emoji-assets/13.0/google-medium/1f604.png!Can someone enlighten me what should I do?
 
Before the save, everything is fine! ("ts" column)

Darvi
SLACK Thread: [https://apache-hudi.slack.com/archives/C4D716NPQ/p1652347742173779]
 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4091
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Affects version(s):
- 0.10.1
- Fix version(s):
- 1.1.0
- Attachment(s):
- 12/May/22 11:37;Darvi77;b97b9e55-58a4-417b-b71c-f6b2d3860da0-0_0-26-1663_20220512111505310.parquet;https://issues.apache.org/jira/secure/attachment/13043566/b97b9e55-58a4-417b-b71c-f6b2d3860da0-0_0-26-1663_20220512111505310.parquet
- 12/May/22 11:38;Darvi77;before-save.png;https://issues.apache.org/jira/secure/attachment/13043565/before-save.png
- 12/May/22 11:37;Darvi77;example-code.txt;https://issues.apache.org/jira/secure/attachment/13043567/example-code.txt

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.