NVIDIA / NVIDIA/cudf

[BUG] Comparing decimal values with positive scale to MAX LONG or MIN LONG produces incorrect results

Open
#9,282 14 comments 0 reactions 1 assignee Claimed by @codereport View on GitHub
bug libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
We have code to check if casting a Decimal to a long will overflow. The default code creates an INT64 scalar value with the min long value in it, and another with the max long value in it. Then it uses binary ops like LESS, GREATER, LESS_EQUAL, and GREATER_EQUAL to see if the values are within the given range. This works generally great for bytes, shorts and ints, but for longs (INT64) it always comes back as true. The values are in the desired range. I have to explicitly create a decimal scalar with a corresponding scale to make this check work. My guess is that both values are being cast to an INT64 before doing the comparison and the result by definition cannot be outside the range of those values.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.