apache / apache/auron

Support unix_date and date_from_unix_date natively

Open
#2,524 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

**Is your feature request related to a problem? Please describe.**

Auron does not currently support native execution of Spark's `unix_date` and `date_from_unix_date` expressions. These expressions fall back to Spark evaluation.

**Describe the solution you'd like**

Add native support for both functions:

- `unix_date(date)` returns an integer day count relative to 1970-01-01.
- `date_from_unix_date(days)` converts that day count back to a date.

Preserve Spark's return types and null behavior. Add tests for zero, positive and negative day counts, integer boundaries, and round-trip conversions. Use column inputs in Spark regression tests and verify that both expressions execute natively.

**Describe alternatives you've considered**

Keep the existing Spark fallback.

**Additional context**

Both functions were introduced in Spark 3.1.0.

```sql
SELECT unix_date(DATE '1970-01-01'); -- 0

SELECT unix_date(DATE '1969-12-31'); -- -1

SELECT date_from_unix_date(0); -- 1970-01-01

SELECT date_from_unix_date(-1); -- 1969-12-31
```

Contributor guide

Open the contributing guide

Research direction

Start by locating existing native date-expression implementations and the Spark regression tests for expression execution. Add coverage for zero, positive and negative day counts, integer boundaries, nulls, and round trips, then verify the column expressions execute natively with Spark-compatible return types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.