NVIDIA / NVIDIA/cudf

[FEA] Filtering/reading statistics of ORC data with legacy TimestampStatistics

Open
#6,158 1 comment 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The ORC statistics reading introduced in #6142 and stats-based ORC filtering in #6116 do not support the legacy version of `TimestampStatistics` in the ORC format. This legacy version uses time that is adjusted to the local timezone that the ORC data was written in. Fortunately, the local timezone is contained in ORC metadata so it is possible for us to support this version.

I'm creating this issue mainly to see if there is a need for this (user/workflow). If there is, the necessary changes would involve an interface between cuDF and libcudf for getting `writerTimezone` from each `StripeFooter`. This `writerTimezone` would be used to decode `minimum` and `maximum` into Python datetimes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.