[FEA] Tighten up promotion when merging with non-equal key column dtypes
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
To date, cudf has attempted to match pandas semantics when matching join keys in a merge. libcudf does not perform merges between mismatching table dtypes. Consequently, the first step of a merge in cudf is to determine a "common" dtype for each pair of columns used as keys in the merge.
The pandas rules are mostly (though not completely since there is some under the table work that happens in the join algorithm) encoded in https://github.com/pandas-dev/pandas/blob/f7c73a5f1aaf0724598e60c0cc5732604ec842a8/pandas/core/reshape/merge.py#L1340
There are a few problems when trying to match these in cudf:
- not all column types in pandas can be represented in cudf (we do not have an `object` column for example)
- it is difficult to unambiguously determine the type promotion rules since they are not written down anywhere
- for example, promotion rules for categorical columns differ depending on whether the categorical is the left or right key.
Moreover, there are other, correctness, problems. The current type promotion rules admit lossy conversions that can result in false positive matches in merges.
Example:
```
left = cudf.DataFrame({"key": [1, 2, 2**53]})
right = cudf.DataFrame({"key": [2**53 + 1, 10]})
right["key"] = right.key.astype("uint64")
left.merge(right, on="key", how="inner")
# key
# 0 9.007199e+15
left
# key
# 0 1
# 1 2
# 2 9007199254740992
right
# key
# 0 9007199254740993
# 1 10
```
Pandas is also susceptible to this, but produces a different wrong result.
I would like to tighten up the rules in cudf, so that it is impossible for the user to get a "surprising" result without some explicit intervention on their behalf. We would also try and match pandas more closely where that is possible, but my preference is to be correct in a subset of cases over dubiously correct in a larger set.
**Describe the solution you'd like**
There are, I think, three levels of things we could do:
1. Push the burden of dtype matching completely on to the user: complain (raise) if merge keys do not match dtypes _exactly_
2. Promote keys whose dtypes allow so safely (without needing to inspect values), and raise for cases where that is not possible. The user can still perform the merge by intentionally casting to matching types. But then they must know that it is safe.
3. Try and match pandas promotions as closely as possible and accept that there might be false positives.
I would like to go for (2). (1) is easiest; (3) is difficult, probably a moving target and can result in false positives without the user explicitly "requesting" them.
With cudf-pandas (2), I think, skates the line between ease of use and correctness reasonably well. We can run as much on the GPU as possible and raise (possibly providing a warning in pandas-compat mode) with fallback to CPU. When using cudf directly, users will hopefully be willing to accept a few more edge cases in the name of consistency.
Concretely this would mean:
- No casting for strings
- No casting for lists
- No casting for structs
- Categoricals:
- if both columns are categorical and match, no casting
- if both columns are categorical and _do not_ match, raise[^1]
- if one column is categorical, unwrap, and go round again[^2]
- No casting for decimals
- No casting for datetimes[^3]
- For numeric types, use a type promotion lattice that has lossless least upper bounds for all types[^4]
For numeric types, that means that we would only promote pairs of types where there exists a wider type whose values are uniquely and unambiguously mapped onto from the narrower types.
For example `(int32, uint32) -> int64` would be allowed, but merging a pair `(int32, uint64)` would raise (since there is no signed 128bit int that we could use). Similarly, we would safely be able to promote `(intX, floatY)` pairs (and similarly with `uintX`) as long as the integer type is 32 or fewer bits wide[^5].
[^1]: I could also be convinced to unwrap and go round again, but that would lose information about the categorical nature of the inputs
[^2]: Pandas behaviour in this case depends on whether the left or right key is categorical (and which merge type it is): it casts the non-categorical to object, and the categorical to its underlying dtype, then imperfectly goes through its matching process again
[^3]: I haven't looked at what pandas does here, but I guess the other thing one could do is promote when one can losslessly convert
[^4]: See, for example https://jax.readthedocs.io/en/latest/jep/9407-type-promotion.html though I disagree with their approach of selecting a "weak" float64 as the least upper bound for `(int64, uint64)`
[^5]: Merging between float and int columns is kind of weird, so I could also be convinced to raise when merging between mismatching numeric kinds.
Contributor guide
Assessment
This issue has not been assessed yet.