[FEA] ORC writer: persist Gregorian calendar mode in the file footer
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Yes. The libcudf ORC writer can encode DATE (`TIMESTAMP_DAYS`) columns, but it does not persist the writer's calendar mode in the standard ORC file footer.
The ORC specification defines `Footer.calendar` as protobuf field 11, with `JULIAN_GREGORIAN` and `PROLEPTIC_GREGORIAN` values:
https://github.com/apache/orc-format/blob/44ca5115b4f3690e1f3d9a3a1100f7b4b1a61fea/src/main/proto/orc/proto/orc_proto.proto#L382-L411
Apache ORC writes this field and uses it when reading. If the field is absent, the reader falls back to `orc.proleptic.gregorian.default`:
- Writer: https://github.com/apache/orc/blob/7f1dd1821bac79ccd017ca73d082bcaba1055b86/java/core/src/java/org/apache/orc/impl/WriterImpl.java#L693-L702
- Reader: https://github.com/apache/orc/blob/7f1dd1821bac79ccd017ca73d082bcaba1055b86/java/core/src/java/org/apache/orc/impl/ReaderImpl.java#L930-L934
At current libcudf main (`9e8e79962d7ced863e209f49da466a23ec0c5819`), the internal ORC `Footer` ends at `writer` (field 9), the protobuf serializer writes only fields 1 through 9, and the public writer options have no calendar setting:
- Footer: https://github.com/NVIDIA/cudf/blob/9e8e79962d7ced863e209f49da466a23ec0c5819/cpp/src/io/orc/orc.hpp#L94-L104
- Serializer: https://github.com/NVIDIA/cudf/blob/9e8e79962d7ced863e209f49da466a23ec0c5819/cpp/src/io/orc/orc.cpp#L303-L315
- Writer options: https://github.com/NVIDIA/cudf/blob/9e8e79962d7ced863e209f49da466a23ec0c5819/cpp/include/cudf/io/orc.hpp#L633-L655
This makes a cuDF-written file non-self-describing with respect to calendar semantics. A standards-compliant reader may treat proleptic day values as legacy Julian/Gregorian values and silently return different dates.
This is a more narrowly scoped writer-side prerequisite for #11691.
### Downstream correctness evidence
The problem was reproduced while fixing NVIDIA/cudf-spark#15470 and NVIDIA/cudf-spark#15496. The downstream Spark reproduction recorded:
```text
RAPIDS Enabled = YES
Plugins Loaded = YES
Spark 2.4 legacy ORC input:
CPU: 1200-01-01
GPU before the downstream read fix: 1200-01-08
```
Writer-side controls recorded during #15496 validation showed that a cuDF-written ORC file without calendar metadata could be interpreted as legacy and return:
```text
written logical value -> value read under legacy/default interpretation
1001-01-01 -> 1000-12-26
1582-10-15 -> 1582-09-30
```
The mismatch also applies when the write requests `orc.proleptic.gregorian=true`: the cuDF writer does not persist that choice, so another reader using the default legacy setting cannot discover the intended calendar from the file.
References and executable downstream coverage:
- Reproduction issue: https://github.com/NVIDIA/cudf-spark/issues/15470
- Correctness fix and writer limitation: https://github.com/NVIDIA/cudf-spark/pull/15496
- Nested STRUCT/ARRAY DATE fallback coverage: https://github.com/NVIDIA/cudf-spark/blob/035284ea10dd5d7e0e4e04f191db365c0238e7c1/tests/src/test/scala/com/nvidia/spark/rapids/OrcQuerySuite.scala#L73-L100
The current downstream mitigation intentionally falls back every ORC write schema containing DATE, including nested DATE values, to the CPU writer:
https://github.com/NVIDIA/cudf-spark/blob/035284ea10dd5d7e0e4e04f191db365c0238e7c1/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuOrcFileFormat.scala#L95-L104
That protects correctness and interoperability, but removes GPU ORC-write acceleration for all DATE-containing schemas, including workloads whose current values are after the 1582 cutover.
**Describe the solution you'd like**
Add standard calendar-mode support to both one-shot and chunked ORC writers.
1. Add a calendar option to `orc_writer_options` and `chunked_orc_writer_options`, preferably as a typed enum representing at least `PROLEPTIC_GREGORIAN` and `JULIAN_GREGORIAN`.
2. Add `CalendarKind` and an optional calendar member to libcudf's internal ORC `Footer`, and serialize it as protobuf field 11.
3. Expose the option through `ai.rapids.cudf.ORCWriterOptions` and the Java/JNI write path.
4. Ensure the encoded DATE values and the footer value agree:
- Proleptic mode: retain the current proleptic day representation and emit `PROLEPTIC_GREGORIAN`.
- Legacy mode: rebase Gregorian day values to the hybrid Julian/Gregorian representation before encoding and emit `JULIAN_GREGORIAN`, or leave this mode unsupported until both conversion and reader behavior are implemented.
5. Teach the libcudf reader to consume or expose the footer calendar mode as needed, so libcudf round trips remain correct for legacy files.
The downstream integration can then be staged safely:
- First re-enable GPU ORC DATE writes only for explicit proleptic mode once the footer field is available.
- Then support legacy mode by recursively rebasing top-level and nested DATE columns on GPU before writing.
Spark RAPIDS already has a GPU Gregorian-to-Julian DATE rebase used by the Parquet writer, so the downstream legacy transformation does not require a new kernel:
https://github.com/NVIDIA/cudf-spark/blob/e0ec3762b5614d79ebb896357e792c9dbc78ce11/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetFileFormat.scala#L391-L409
### Suggested acceptance coverage
- libcudf ORC write -> Apache ORC/Spark CPU read and libcudf read.
- `PROLEPTIC_GREGORIAN`, `JULIAN_GREGORIAN`, and missing/default behavior.
- Dates before, at, and after the October 1582 cutover, including `1001-01-01`, `1200-01-01`, `1582-10-04`, `1582-10-15`, and a modern control.
- Top-level DATE, STRUCT, LIST, nulls, empty input, one-shot writer, and chunked writer.
- Footer inspection through Apache ORC `Reader.writerUsedProlepticGregorian()`.
- DATE statistics and predicate-pushdown correctness after any legacy rebase.
**Describe alternatives you've considered**
- **CPU fallback:** correct and currently used by cudf-spark, but it can materially reduce ORC write performance.
- **Force every reader to use `orc.proleptic.gregorian.default=true`:** works only in a fully controlled reader environment and leaves the file non-self-describing; it is not a general interoperability fix.
- **Write a custom key/value metadata entry:** standard ORC readers consult `Footer.calendar`, not a private key, so this does not solve the problem.
- **Post-process the ORC footer:** brittle because the footer is encoded/compressed and its lengths must be rewritten; it also adds another I/O pass.
- **Allow GPU writes only when values are known to be post-cutover:** the planner currently sees the schema, not every value, and this still leaves the file's calendar contract ambiguous.
**Additional context**
- Existing broad correctness issue: #11691
- Downstream compatibility issue: https://github.com/NVIDIA/cudf-spark/issues/131
- The open writer-timezone work in #23449 is a useful API/JNI plumbing precedent, but it does not add ORC calendar metadata.
Contributor guide
Research direction
Start with the internal Footer and serializer in cpp/src/io/orc/orc.hpp and cpp/src/io/orc/orc.cpp, then trace the one-shot and chunked options in cpp/include/cudf/io/orc.hpp and the Java/JNI write path. Review existing ORC writer and reader tests before deciding how option plumbing and DATE handling are covered. Done means both writers expose the selected calendar and interoperability tests cover footer inspection, round trips, cutover dates, nested DATE values, and missing/default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100