apache / apache/datafusion-comet
[Feature] Support Spark expression: make_dt_interval
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
Comet does not currently support the Spark `make_dt_interval` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
The `MakeDTInterval` expression creates a day-time interval value from separate day, hour, minute, and second components. This expression is used to construct `DayTimeIntervalType` values programmatically by combining individual time unit values into a single interval representation.
Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
## Describe the potential solution
### Spark Specification
**Syntax:**
```sql
make_dt_interval(days, hours, minutes, seconds)
make_dt_interval(days, hours, minutes)
make_dt_interval(days, hours)
make_dt_interval(days)
make_dt_interval()
```
**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| `days` | IntegerType | Number of days in the interval (optional, defaults to 0) |
| `hours` | IntegerType | Number of hours in the interval (optional, defaults to 0) |
| `minutes` | IntegerType | Number of minutes in the interval (optional, defaults to 0) |
| `seconds` | DecimalType(MAX_LONG_DIGITS, 6) | Number of seconds including microsecond precision (optional, defaults to 0) |
**Return Type:** Returns a `DayTimeIntervalType()` representing the constructed interval.
**Supported Data Types:**
- **days**: Integer values
- **hours**: Integer values
- **minutes**: Integer values
- **seconds**: Decimal values with up to 6 decimal places for microsecond precision
**Edge Cases:**
- **Null handling**: Expression is null-intolerant (`nullIntolerant = true`), meaning if any input is null, the result is null
- **Default values**: Missing parameters default to 0 (literal values)
- **Precision handling**: Seconds parameter uses DecimalType with 6 decimal places to preserve microsecond precision
- **Overflow behavior**: Delegates to `IntervalUtils.makeDayTimeInterval()` for overflow validation and error handling
- **Error context**: Includes query context information for meaningful error messages when interval construction fails
**Examples:**
```sql
-- Create a 5-day, 3-hour, 30-minute, 45.5-second interval
SELECT make_dt_interval(5, 3, 30, 45.5);
-- Create a 2-day interval
SELECT make_dt_interval(2);
-- Create a 1-day, 12-hour interval
SELECT make_dt_interval(1, 12);
-- Create an empty interval
SELECT make_dt_interval();
```
```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
// Create interval from literal values
df.select(expr("make_dt_interval(5, 3, 30, 45.5)"))
// Create interval from column values
df.select(expr("make_dt_interval(day_col, hour_col, min_col, sec_col)"))
```
### Implementation Approach
See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
## Additional context
**Difficulty:** Large
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.MakeDTInterval`
**Related:**
- `MakeYMInterval` - Creates year-month intervals
- `IntervalUtils` - Utility class for interval operations
- `DayTimeIntervalType` - The data type returned by this expression
- `Extract` - Extracts components from interval values
---
*This issue was auto-generated from Spark reference documentation.*
Contributor guide
Assessment
This issue has not been assessed yet.