apache / apache/datafusion-comet
[Feature] Support Spark expression: divide_dt_interval
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `divide_dt_interval` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
The `DivideDTInterval` expression divides a day-time interval by a numeric value, returning a new day-time interval. This operation performs division with `HALF_UP` rounding mode and includes comprehensive overflow and divide-by-zero checks to ensure safe arithmetic operations.
Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
day_time_interval / numeric_value
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| `interval` | `DayTimeIntervalType` | The day-time interval to be divided |
| `num` | `NumericType` | The numeric divisor (integral, decimal, or fractional types) |
**Return Type:** `DayTimeIntervalType()` - Returns a day-time interval representing the result of the division operation.
**Supported Data Types:**
- **Left operand (interval)**: `DayTimeIntervalType` only
- **Right operand (num)**: All numeric types including:
- `IntegralType` (Byte, Short, Int, Long)
- `DecimalType`
- `FractionalType` (Float, Double)
**Edge Cases:**
- **Null handling**: Expression is null-intolerant (`nullIntolerant = true`) - returns null if either operand is null
- **Divide by zero**: Throws `QueryExecutionErrors` when divisor is zero
- **Overflow behavior**:
- Checks for `Long.MinValue / -1` overflow condition
- Throws `QueryExecutionErrors.overflowInIntegralDivideError` on overflow
- Uses `longValueExact()` for decimal operations to detect overflow
- **Precision**: All division operations use `HALF_UP` rounding mode for consistent behavior
**Examples:**
```sql
-- Divide a day-time interval by an integer
SELECT INTERVAL '2 12:30:45.123' DAY TO SECOND / 2;
-- Result: INTERVAL '1 06:15:22.561500' DAY TO SECOND
-- Divide by decimal value
SELECT INTERVAL '5 00:00:00' DAY TO SECOND / 2.5;
-- Result: INTERVAL '2 00:00:00' DAY TO SECOND
```
```scala
// Example DataFrame API usage
import org.apache.spark.sql.functions._
// Divide interval column by numeric literal
df.select(col("day_time_interval") / lit(3))
// Divide interval by another numeric column
df.select(col("day_time_interval") / col("divisor"))
```
### Implementation Approach
See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
## Additional context
**Difficulty:** Medium
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.DivideDTInterval`
**Related:**
- `DivideYMInterval` - Division operation for year-month intervals
- `MultiplyDTInterval` - Multiplication of day-time intervals
- `IntervalDivide` - Base trait for interval division operations
- `BinaryExpression` - Parent class for binary operations
---
*This issue was auto-generated from Spark reference documentation.*
Contributor guide
Assessment
This issue has not been assessed yet.