llvm / llvm/llvm-project

[clang] Casting unsigned long _Fract to int emits out-of-bounds shift

Open
#221,207 1 comment 0 reactions 0 assignees View on GitHub
clang:codegen
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Casting `unsigned long _Fract` to an integer emits an out-of-bounds shift by the bitwidth in `FixedPointBuilder`, leading to undefined behavior in LLVM IR.

When downscaling in `FixedPointBuilder::Convert` (`llvm/include/llvm/IR/FixedPointBuilder.h`), it creates a right shift by `SrcScale - DstScale`:

```cpp
unsigned Shift = SrcScale - DstScale;
if (SrcIsSigned)
Result = B.CreateAShr(Result, Shift, "downscale");
else
Result = B.CreateLShr(Result, Shift, "downscale");

```

For `unsigned long _Fract`, `SrcScale` is 32 (its full bitwidth) and `DstScale` is 0. This emits `lshr i32 %x, 32`, which is an invalid shift by the operand bitwidth.

For example:

```cpp
int to_int(unsigned long _Fract x) {
return (int)x;
}
```

Compiling with `clang++ -ffixed-point -O0 -S -emit-llvm` gives:

```llvm
define dso_local noundef i32 @_Z6to_intDRm(i32 noundef %0) {
...
%3 = load i32, ptr %2, align 4
%4 = lshr i32 %3, 32
ret i32 %4
}
```

Since `unsigned long _Fract` values are in [0, 1), converting to an integer should evaluate to 0.

This was uncovered by a test failure in LLVM-libc's `countlsulr_test` (`libc/test/src/stdfix/CountlsTest.h`). In `CountlsTest.h`, the check `if (10 <= static_cast(max))` became undefined due to the invalid shift. At runtime the backend branched on uninitialized register state, which occasionally took the branch and failed with `Expected: FXRep::INTEGRAL_LEN - 4 (-4), To be equal to: func(10) (0)`.

In `FixedPointBuilder::Convert`, if `SrcScale - DstScale >= ResultWidth`, downscaling an unsigned value should return constant zero instead of emitting an invalid shift.

Assisted-by: Automated tooling, human reviewed

Contributor guide

Open the contributing guide

Research direction

Start in llvm/include/llvm/IR/FixedPointBuilder.h, focusing on FixedPointBuilder::Convert and its downscaling shift logic. Then inspect llvm/libc/test/src/stdfix/CountlsTest.h and run the relevant countlsulr_test. Done means converting unsigned long _Fract values such as 10 produces defined LLVM IR and the test no longer observes an invalid shift or incorrect result.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.