Decide properly on semantics of empty `Channel`s in cudf-polars streaming backend
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
A `Channel` across multiple ranks in cudf-polars is a logical representation of a `cudf::table`. The full table may be reconstructed by the concatenation of all chunks in the (ordered) `Channel` in rank order.
In some cases, a rank's channel may be _empty_. That is, it contains no chunks at all. For example, suppose we take the slice of the `head` of a table. It is likely that all ranks other than rank-0 will have no rows.
Most of the time, the code works fine with a completely empty channel, but occasionally if we're not doing communication we end up branching and doing something like:
```
chunks = await collect_all_chunks_from_channel(ch)
if chunks:
table = await evaluate_batch(chunks, ...)
else:
table = empty_table(...)
```
This is for (probably) two reasons:
1. `evaluate_batch` does `cudf::concatenate` and that wants at least one table to concatenate
2. The downstream consumer is expecting at least one chunk so it can obtain the correct schema
This is currently done in an ad-hoc manner based on whether something happens to break or not. We should instead specify what semantics we support and then do everything consistently.
cc @rjzamora, @madsbk
Contributor guide
Assessment
This issue has not been assessed yet.