NVIDIA / NVIDIA/cudf

Decide properly on semantics of empty `Channel`s in cudf-polars streaming backend

Open
#22,472 2 comments 0 reactions 0 assignees View on GitHub
cudf-polars proposal
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

A `Channel` across multiple ranks in cudf-polars is a logical representation of a `cudf::table`. The full table may be reconstructed by the concatenation of all chunks in the (ordered) `Channel` in rank order.

In some cases, a rank's channel may be _empty_. That is, it contains no chunks at all. For example, suppose we take the slice of the `head` of a table. It is likely that all ranks other than rank-0 will have no rows.

Most of the time, the code works fine with a completely empty channel, but occasionally if we're not doing communication we end up branching and doing something like:

```
chunks = await collect_all_chunks_from_channel(ch)
if chunks:
table = await evaluate_batch(chunks, ...)
else:
table = empty_table(...)
```

This is for (probably) two reasons:

1. `evaluate_batch` does `cudf::concatenate` and that wants at least one table to concatenate
2. The downstream consumer is expecting at least one chunk so it can obtain the correct schema

This is currently done in an ad-hoc manner based on whether something happens to break or not. We should instead specify what semantics we support and then do everything consistently.

cc @rjzamora, @madsbk

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.