apache / apache/datafusion-comet

Add compatible timezone support for date_format expression

Open
#3,202 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Description

The `date_format` expression was added in PR #3201, but currently only supports UTC timezone with full compatibility. Non-UTC timezones are marked as `Incompatible` and fall back to Spark by default.

This issue tracks adding proper timezone conversion support so that `date_format` can be fully compatible with Spark for all timezones.

## Current Behavior

- UTC timezone: `Compatible()` - runs natively in Comet
- Non-UTC timezones: `Incompatible()` - falls back to Spark by default
- Users can enable non-UTC with `spark.comet.expr.DateFormatClass.allowIncompatible=true` but results may differ from Spark

## Desired Behavior

All timezones should be `Compatible()` and produce results identical to Spark.

## Technical Details

The current implementation uses DataFusion's `to_char` function which formats timestamps without timezone conversion. Spark's `date_format` applies the session timezone when formatting.

Possible approaches:
1. Convert the timestamp to the target timezone before calling `to_char`
2. Use a timezone-aware formatting function if available in DataFusion
3. Implement custom Rust logic to handle timezone conversion

## Related

- PR #3201 - Initial date_format implementation
- Issue #2649 - Similar timezone issue with `date_trunc`

---
> **Note:** This issue was generated with AI assistance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.