[clang] Casting unsigned long _Fract to int emits out-of-bounds shift
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
Casting `unsigned long _Fract` to an integer emits an out-of-bounds shift by the bitwidth in `FixedPointBuilder`, leading to undefined behavior in LLVM IR.
When downscaling in `FixedPointBuilder::Convert` (`llvm/include/llvm/IR/FixedPointBuilder.h`), it creates a right shift by `SrcScale - DstScale`:
```cpp
unsigned Shift = SrcScale - DstScale;
if (SrcIsSigned)
Result = B.CreateAShr(Result, Shift, "downscale");
else
Result = B.CreateLShr(Result, Shift, "downscale");
```
For `unsigned long _Fract`, `SrcScale` is 32 (its full bitwidth) and `DstScale` is 0. This emits `lshr i32 %x, 32`, which is an invalid shift by the operand bitwidth.
For example:
```cpp
int to_int(unsigned long _Fract x) {
return (int)x;
}
```
Compiling with `clang++ -ffixed-point -O0 -S -emit-llvm` gives:
```llvm
define dso_local noundef i32 @_Z6to_intDRm(i32 noundef %0) {
...
%3 = load i32, ptr %2, align 4
%4 = lshr i32 %3, 32
ret i32 %4
}
```
Since `unsigned long _Fract` values are in [0, 1), converting to an integer should evaluate to 0.
This was uncovered by a test failure in LLVM-libc's `countlsulr_test` (`libc/test/src/stdfix/CountlsTest.h`). In `CountlsTest.h`, the check `if (10 <= static_cast(max))` became undefined due to the invalid shift. At runtime the backend branched on uninitialized register state, which occasionally took the branch and failed with `Expected: FXRep::INTEGRAL_LEN - 4 (-4), To be equal to: func(10) (0)`.
In `FixedPointBuilder::Convert`, if `SrcScale - DstScale >= ResultWidth`, downscaling an unsigned value should return constant zero instead of emitting an invalid shift.
Assisted-by: Automated tooling, human reviewed
Contributor guide
Research direction
Start in llvm/include/llvm/IR/FixedPointBuilder.h, focusing on FixedPointBuilder::Convert and its downscaling shift logic. Then inspect llvm/libc/test/src/stdfix/CountlsTest.h and run the relevant countlsulr_test. Done means converting unsigned long _Fract values such as 10 produces defined LLVM IR and the test no longer observes an invalid shift or incorrect result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100