apache / apache/auron

Add non-UTC timestamp coverage for native datediff

Open Beginner friendly
#2,510 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Is your feature request related to a problem? Please describe.**

PR #2502 added native support for `datediff(endDate, startDate)`, but the current regression coverage only uses `DATE` inputs.

Spark also accepts `TIMESTAMP` inputs and converts them to dates using the session time zone. The native path should cover cases where the local date differs from the UTC date.

**Describe the solution you'd like**

Add a regression case to `AuronFunctionSuite` that:

- uses `TIMESTAMP` columns;
- sets `spark.sql.session.timeZone` to a non-UTC zone such as `America/Los_Angeles`;
- uses values around a local/UTC date boundary;
- verifies that the result matches Spark and that the native operator is used.

For example:

```sql
CREATE TABLE t1(end_ts TIMESTAMP, start_ts TIMESTAMP) USING parquet;

INSERT INTO t1 VALUES
(TIMESTAMP '1970-01-01 16:30:00', TIMESTAMP '1970-01-01 00:30:00');

SELECT datediff(end_ts, start_ts) FROM t1;
```

With America/Los_Angeles as the session time zone, the expected result is 0.

**Describe alternatives you've considered**

Keep the existing DATE-only coverage. The implementation remains covered for its core calculation, but the timestamp-to-date conversion path is not protected against regressions.

**Additional context**

This is a follow-up to #2501 and #2502.
The issue only tracks regression coverage and does not assume that the current implementation has a correctness problem.

Contributor guide

Open the contributing guide

Research direction

Start with the existing datediff coverage in AuronFunctionSuite and inspect how it verifies native operator use. Add a TIMESTAMP case using a non-UTC spark.sql.session.timeZone and values around a date boundary, then run the suite to confirm the native result matches Spark and returns 0 for the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark, sql
Domain
data-engineering, testing
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.