[FEA] Better handling for nullable columns that do not have any null element
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Currently, many cudf's APIs (such as `copy_bitmask` or `valid_if` and many others) always generate an output bitmask if the input columns are nullable. However, the input columns may not have any null element. Thus, it may be a waste of time and memory to generate such a bitmask containing all valid bit. So, a good practice is to avoid generating a bitmask if the input does not contain any null element. Some APIs have already followed this practice, but not all.
However, if not generating a bitmask for the output, there may be potential issues emerging from the inconsistency between the input and output. Consider an example that an algorithm processing data through several stages. One of the intermediate stages cuts out the bitmask from its output, but all the subsequent stages continue to use the nullable information from the first input. In such cases, the system may crash, or some stages throw an exception, or the final result is undefined.
So, the question here is, should we keep the consistency between the input and output column? (i.e., if the input column has a bitmask, should we always generate a bitmask for the output column even if it doesn't have any null?)
---------
I have observed a lot of bugs that emerged due to mixing `nullable()` and `null_count()` in the current cudf implementation. Thus, I think a good way to avoid (maybe) all potential issues is to totally remove the `nullable()` API and use `null_count()` all the time. Maybe this is a little bit more expensive for the users to execute the code, but it could definitely eliminate all the headaches from having unexpected bugs and make developers/users much happier.
Contributor guide
Assessment
This issue has not been assessed yet.