apache / apache/datafusion-comet
[Feature] Support Spark expression: multiply_ym_interval
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `multiply_ym_interval` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
The `MultiplyYMInterval` expression multiplies a year-month interval by a numeric value, returning a new year-month interval. This expression supports various numeric data types and handles precision conversions appropriately for each type.
Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
year_month_interval * numeric_value
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| `interval` | YearMonthIntervalType | The year-month interval to be multiplied |
| `num` | NumericType | The numeric multiplier (byte, short, int, long, float, double, or decimal) |
**Return Type:** `YearMonthIntervalType()` - Returns a year-month interval representing the multiplication result.
**Supported Data Types:**
- **Interval**: YearMonthIntervalType only
- **Numeric**: All numeric types including:
- Integer types: ByteType, ShortType, IntegerType, LongType
- Floating-point types: FloatType, DoubleType
- DecimalType (arbitrary precision)
**Edge Cases:**
- **Null handling**: Returns null if either operand is null (nullIntolerant = true)
- **Overflow behavior**:
- Integer multiplication uses `Math.multiplyExact()` throwing ArithmeticException on overflow
- Long results converted using `Math.toIntExact()` throwing exception if out of int range
- Decimal results use `intValueExact()` throwing exception if fractional part exists after rounding
- **Floating-point precision**: Uses HALF_UP rounding mode for consistent behavior
- **Zero multiplication**: Results in zero-month interval (valid)
**Examples:**
```sql
-- Multiply interval by integer
SELECT INTERVAL '2-6' YEAR TO MONTH * 3; -- Results in '7-6' (7 years 6 months)
-- Multiply interval by decimal
SELECT INTERVAL '1-0' YEAR TO MONTH * 2.5; -- Results in '2-6' (2 years 6 months)
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(col("year_month_interval") * lit(2))
df.select(col("year_month_interval") * col("multiplier"))
```
### Implementation Approach
See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.MultiplyYMInterval`
**Related:**
- `MultiplyDTInterval` - For day-time interval multiplication
- `DivideYMInterval` - For year-month interval division
- `AddYMInterval` - For year-month interval addition
- `SubtractYMInterval` - For year-month interval subtraction
---
*This issue was auto-generated from Spark reference documentation.*
Contributor guide
Assessment
This issue has not been assessed yet.