deepseek-ai / deepseek-ai/DeepGEMM

fix: operator precedence bug in pack_ue8m0_to_int assertion (mantissa check always passes)

Open
#309 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
7.8k
Forks
1.3k
Avg merge
3d 7h
Merged PRs (30d)
3

Description

In `deep_gemm/utils/math.py`, the `pack_ue8m0_to_int` function has an operator precedence bug that makes the validation assertion ineffective.

## Bug

```python
def pack_ue8m0_to_int(x: torch.Tensor):
assert x.dtype == torch.float and x.size(-1) % 4 == 0
assert (x.view(torch.int) & ((1 << 23) - 1) == 0).all() # BUG: wrong precedence
```

In Python, `==` has higher precedence than `&`. So the expression is parsed as:
```
x.view(torch.int) & (((1 << 23) - 1) == 0)
```

`((1 << 23) - 1) == 0` evaluates to `False` (which is `0` numerically), then `x.view(torch.int) & 0` gives all zeros, and `.all()` on all zeros returns `True`.

This means the assertion **always passes** regardless of whether the mantissa bits are actually zero, so it silently accepts invalid inputs instead of catching them.

## Fix

Add parentheses to ensure the AND operation happens before the equality check:

```python
assert ((x.view(torch.int) & ((1 << 23) - 1)) == 0).all()
```

This correctly checks that all mantissa bits (lower 23 bits) are zero, which is the intended behavior for UE8M0 format values.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.