NVIDIA / NVIDIA/cudf

[FEA] Improve cudf tests for hash_partition

Open
#7,819 2 comments 0 reactions 0 assignees View on GitHub
feature request Python tests
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

As `cudf::hash_partition` now can use a hash function other than Murmur3 or take a custom seed value, I think the tests should be improved for better coverage.

**Describe the solution you'd like**

As described in [this comment](https://github.com/rapidsai/cudf/pull/7771#issuecomment-811692022), two key properties of `cudf::hash_partition` are

1. Output partitions are disjoint; and
2. Rows having the same key are always assigned the same partition id, even when they appear in different dataframes (when the number of partitions is fixed).

We need to check these properties for some combinations of hash function types and seed values. Property 1 is checked in [this Python test](https://github.com/rapidsai/cudf/blob/b9415cae362882380e14b5baa6af49275021356e/python/cudf/cudf/tests/test_dataframe.py#L1106-L1135), but only with the default hash function and seed value, and I believe there are no tests for property 2 yet.

Additionally, we should remove [this test](https://github.com/rapidsai/cudf/blob/branch-0.19/python/cudf/cudf/tests/test_dataframe.py#L1181-L1200). The test is comparing a hard-coded value with the result of `hash_partition` for a test input, but this is over-constraining, as `hash_partition` in general has no guarantee on the order in which the keys appear in the output partitions. If we later decide to change the `row_hasher` in any way, this test will start to fail.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.