apache / apache/arrow-rs

[EPIC] Optimize the `coalesece` kernel (`BatchCoalescer`)

Open
#7,761 0 comments 2 reactions 1 assignee Claimed by @alamb View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

This ticket trans improvements to the [`BatchCoalescer`](https://github.com/apache/arrow-rs/blob/2b40d1dfc35862ff350a40dfbc66f8a14f4eea31/arrow-select/src/coalesce.rs#L38-L122) for the usecase described in https://github.com/apache/arrow-rs/issues/6692

That is
```

┌────────────────────┐ Filter
│ │ ┌────────────────────┐ Coalesce
│ │ ─ ─ ─ ─ ─ ─ ▶ │ RecordBatch │ Batches
│ RecordBatch │ │ num_rows = 234 │─ ─ ─ ─ ─ ┐
│ num_rows = 8000 │ └────────────────────┘
│ │ │
│ │ ┌────────────────────┐
└────────────────────┘ │ │ │
┌────────────────────┐ ┌────────────────────┐ │ │
│ │ Filter │ │ │ │ │
│ │ │ RecordBatch │ ─ ─ ─ ─ ─ ▶│ │
│ RecordBatch │ ─ ─ ─ ─ ─ ─ ▶ │ num_rows = 500 │─ ─ ─ ─ ─ ┐ │ │
│ num_rows = 8000 │ │ │ │ RecordBatch │
│ │ │ │ └ ─ ─ ─ ─ ─▶│ num_rows = 8000 │
│ │ └────────────────────┘ │ │
└────────────────────┘ │ │
... ─ ─ ─ ─ ─ ▶│ │
... ... │ │ │
│ │
┌────────────────────┐ │ └────────────────────┘
│ │ ┌────────────────────┐
│ │ Filter │ │ │
│ RecordBatch │ │ RecordBatch │
│ num_rows = 8000 │ ─ ─ ─ ─ ─ ─ ▶ │ num_rows = 333 │─ ─ ─ ─ ─ ┘
│ │ │ │
│ │ └────────────────────┘
└────────────────────┘

FilterExec RepartitonExec copies the data
creates output batches with copies *again* to form final large
of the matching rows (calls take() RecordBatches
to make a copy)

```

This ticket tracks additional follow on work:
- [x] https://github.com/apache/arrow-rs/issues/7763
- [ ] https://github.com/apache/arrow-rs/issues/7764
- [x] https://github.com/apache/arrow-rs/issues/7762
- [x] https://github.com/apache/arrow-rs/issues/9143
- [ ] https://github.com/apache/arrow-rs/issues/7765
- [x] https://github.com/apache/arrow-rs/issues/9136
- [ ] https://github.com/apache/arrow-rs/issues/8850

**Additional context**
- the use case is described in detail here https://github.com/apache/arrow-rs/issues/6692

**Additional context**
- See https://github.com/apache/arrow-rs/issues/6692 for more details

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.