NVIDIA / NVIDIA/cudf

[BUG] split/slice APIs do not align with partitioning APIs

Open
#4,607 8 comments 0 reactions 0 assignees View on GitHub
bug libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Partitioning APIs that partition a table into `n` partitions, like [`hash_partition`](https://github.com/rapidsai/cudf/blob/branch-0.14/cpp/include/cudf/partitioning.hpp#L43) or [`round_robin_partition`](https://github.com/rapidsai/cudf/blob/branch-0.14/cpp/include/cudf/partitioning.hpp#L185), return a single table and a vector of `n+1` offsets that points to the beginning of each partition and where the size of any partition `i` can be determined by `offsets[i+1] - offsets[i]`.

For example:
```
partitioned_table = {7}, {}, {3, 8, 9}, {42};
offsets = [0, 1, 1, 4, 5]
```

I would expect to be able to trivially pass the output of a partitioning API into an API like [`split`](https://github.com/rapidsai/cudf/blob/branch-0.13/cpp/include/cudf/copying.hpp#L389) or [`slice`](https://github.com/rapidsai/cudf/blob/branch-0.13/cpp/include/cudf/copying.hpp#L358) in order to get a vector of zero-copy `table_view`s for each partition.

However, this is not possible because the expected inputs for `split` or `slice` are incompatible with the `offsets` vector returned from a partitioning API.

`slice` expects a vector of index pairs:
```
input: [{10, 12, 14, 16, 18, 20, 22, 24, 26, 28},
{50, 52, 54, 56, 58, 60, 62, 64, 66, 68}]
indices: {1, 3, 5, 9, 2, 4, 8, 8}
output: [{{12, 14}, {20, 22, 24, 26}, {14, 16}, {}},
{{52, 54}, {60, 22, 24, 26}, {14, 16}, {}}]
```

`split` expects a vector of the split points:
```
input: {10, 12, 14, 16, 18, 20, 22, 24, 26, 28}
splits: {2, 5, 9}
output: {{10, 12}, {14, 16, 18}, {20, 22, 24, 26}, {28}}
```

Neither of these are trivially compatible with the output of a partitioning API.

`split` is the closest. You can obtain the `splits` vector from the `offsets` vector by dropping the first and last element from `offsets`. However, that is inconvenient.

**Expected behavior**

There should be an API that allows naively passing in the vector of offsets from a partitioning API and it returns a vector of zero-copy views for each partition.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.