apache / apache/datafusion

Optimize `date_part` Minute by avoiding unnecessary computation

Open
#14,043 3 comments 0 reactions 1 assignee Claimed by @samsond View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Open an issue to track the status of this optimization

Related
https://github.com/apache/datafusion/issues/13449
https://github.com/apache/arrow-rs/issues/6746

### Describe the solution you'd like

I guess there are some changes in `arrow-rs` left to do.

### Describe alternatives you've considered

_No response_

### Additional context

```rust
impl ExtractDatePartExt for PrimitiveArray {
fn date_part(&self, part: DatePart) -> Result {
// TimestampSecond only encodes number of seconds, so these will always be 0
let array =
if let DatePart::Millisecond | DatePart::Microsecond | DatePart::Nanosecond = part {
Int32Array::new(vec![0; self.len()].into(), self.nulls().cloned())
} else if let Some(tz) = get_tz(self.data_type())? {
let map_func = get_date_time_part_extract_fn(part);
self.unary_opt(|d| {
timestamp_s_to_datetime(d)
.map(|c| Utc.from_utc_datetime(&c).with_timezone(&tz))
.map(map_func)
})
} else {
let map_func = get_date_time_part_extract_fn(part);
self.unary_opt(|d| timestamp_s_to_datetime(d).map(map_func))
};
Ok(array)
}
}
```

If I remember correctly, we need to switch `timestamp_s_to_datetime` to `timestamp_s_to_time` and extract the data from `Minute`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.