apache / apache/datafusion-comet

[Feature] Support Spark expression: timestamp_add_ym_interval

Open
#3,115 0 comments 0 reactions 0 assignees View on GitHub
enhancement temporal expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.

Comet does not currently support the Spark `timestamp_add_ym_interval` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.

The `TimestampAddYMInterval` expression adds a year-month interval to a timestamp value. This operation is timezone-aware and handles both `TimestampType` and `TimestampNTZType` inputs while preserving the original timestamp data type.

Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.

## Describe the potential solution

### Spark Specification

**Syntax:**
```sql
timestamp_column + INTERVAL 'value' YEAR TO MONTH
```

```scala
// DataFrame API usage
col("timestamp_column") + expr("INTERVAL '2-3' YEAR TO MONTH")
```

**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| timestamp | Expression | The timestamp expression to add the interval to |
| interval | Expression | The year-month interval expression to add |
| timeZoneId | Option[String] | Optional timezone identifier for timezone-aware operations |

**Return Type:** Returns the same data type as the input timestamp expression (`TimestampType` or `TimestampNTZType`).

**Supported Data Types:**
- Input timestamp: `AnyTimestampType` (`TimestampType` or `TimestampNTZType`)
- Input interval: `YearMonthIntervalType`

**Edge Cases:**
- **Null handling**: Returns null if either timestamp or interval input is null (null intolerant)
- **Timezone handling**: Uses session timezone for `TimestampType` and UTC for `TimestampNTZType`
- **Month overflow**: Handles month arithmetic that crosses year boundaries correctly
- **Day adjustment**: May adjust day values when adding months to dates like January 31st + 1 month

**Examples:**
```sql
-- Add 2 years and 3 months to a timestamp
SELECT timestamp_col + INTERVAL '2-3' YEAR TO MONTH FROM events;

-- Add 1 year to current timestamp
SELECT current_timestamp() + INTERVAL '1' YEAR;
```

```scala
// DataFrame API usage
import org.apache.spark.sql.functions._

df.select(col("created_at") + expr("INTERVAL '1-6' YEAR TO MONTH"))

// Using interval column
df.select(col("timestamp_col") + col("interval_col"))
```

### Implementation Approach

See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.

1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)

## Additional context

**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.TimestampAddYMInterval`

**Related:**
- `DateAddYMInterval` - Adds year-month intervals to date values
- `TimestampAddDTInterval` - Adds day-time intervals to timestamps
- `DateTimeUtils.timestampAddMonths()` - Underlying implementation method

---
*This issue was auto-generated from Spark reference documentation.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.