facebookresearch / facebookresearch/dlrm
data_utils.py fails with Numpy 2.0's new handling of overflow error
- Dominant language
- Python
- Stars
- 4.1k
- Forks
- 859
- PR merge metrics
- No merged PRs in 30d
Description
Processing Criteo Kaggle dataset through data_utils.py fails with Numpy 2.
This is because Numpy 2 does not allow out-of-bound coversion of python integers (there was a deprecation warning from [v1.24](https://numpy.org/devdocs/release/1.24.0-notes.html#conversion-of-out-of-bound-python-integers), and from [v2](https://github.com/numpy/numpy/issues/26596) it throws an error), which data_utils.py uses.
In Numpy 1.23, `np.array(2162322587, dtype=np.int32)` gives:
`array(-2132644709, dtype=int32)`
In Numpy 2.0, `np.array(2162322587, dtype=np.int32)` gives
`OverflowError: Python integer 2162322587 out of bounds for int32`
For example, [this line](https://github.com/facebookresearch/dlrm/blob/64063a359596c72a29c670b4fcc9450bb342e764/data_utils.py#L1022) causes an issue, because some sparse features of Criteo dataset is larger than 0x7FFFFFFF and cause an overflow in int32. Changing `np.int32` to `np.uint32` seems to fix the issue, but I haven't tested the entire codebase thoroughly, so there might be other similar bugs (hence, submitting this as an issue instead of submitting a PR).
How to reproduce: run `bench/dlrm_s_criteo_kaggle.sh` with Numpy 2.x.
Contributor guide
Research direction
Start with data_utils.py at the linked line around 1022 and reproduce the failure by running bench/dlrm_s_criteo_kaggle.sh with Numpy 2.x. Search data_utils.py for similar int32 conversions affecting sparse Criteo features, then verify that Criteo Kaggle processing completes without the Numpy overflow error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100