apache / apache/datafusion-comet
Add compatible timezone support for date_format expression
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Description
The `date_format` expression was added in PR #3201, but currently only supports UTC timezone with full compatibility. Non-UTC timezones are marked as `Incompatible` and fall back to Spark by default.
This issue tracks adding proper timezone conversion support so that `date_format` can be fully compatible with Spark for all timezones.
## Current Behavior
- UTC timezone: `Compatible()` - runs natively in Comet
- Non-UTC timezones: `Incompatible()` - falls back to Spark by default
- Users can enable non-UTC with `spark.comet.expr.DateFormatClass.allowIncompatible=true` but results may differ from Spark
## Desired Behavior
All timezones should be `Compatible()` and produce results identical to Spark.
## Technical Details
The current implementation uses DataFusion's `to_char` function which formats timestamps without timezone conversion. Spark's `date_format` applies the session timezone when formatting.
Possible approaches:
1. Convert the timestamp to the target timezone before calling `to_char`
2. Use a timezone-aware formatting function if available in DataFusion
3. Implement custom Rust logic to handle timezone conversion
## Related
- PR #3201 - Initial date_format implementation
- Issue #2649 - Similar timezone issue with `date_trunc`
---
> **Note:** This issue was generated with AI assistance.
Contributor guide
Assessment
This issue has not been assessed yet.