huggingface / huggingface/datasets

`Dataset.select` raises a raw OverflowError on in-range negative indices that its own validation allows

Open Beginner friendly
#8,475 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`Dataset.select` validates indices with [`_check_valid_indices_value`](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_dataset.py#L636-L638), which implements Python-style negative indexing: it only raises for a negative index when `index + size < 0`, so in-range negative indices like `-1` pass validation. But [`_select_with_indices_mapping`](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_dataset.py#L4577-L4586) then converts them with `pa.array(indices, type=pa.uint64())`, which crashes with a raw pyarrow `OverflowError`.

So the two layers disagree on whether negative indices are supported:

- out-of-range negative indices get a clean `IndexError`, while in-range ones crash with `OverflowError: can't convert negative value to unsigned int`;
- `ds[[0, -1]]` (`__getitem__` with a list) has supported negative indices since #668/#679, and #8412 is currently fixing the same family in the formatting path — `select` remains the entry point that crashes.

Negative indices can't reach the contiguous fast path either (`select` requires `indices.start >= 0` / `start >= 0` there), so every negative index ends up in `_select_with_indices_mapping` and crashes.

### Steps to reproduce the bug

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": list(range(10))})
ds[[0, -1]] # {'x': [0, 9]} — getitem supports negative indices
ds.select([-11]) # IndexError: Index -11 out of range for dataset of size 10. (clean)
ds.select([-1]) # OverflowError: can't convert negative value to unsigned int
ds.select(range(-3, 0)) # same OverflowError
ds.select([9, 8, 7]).select([-1]) # same OverflowError with an indices mapping
```

### Expected behavior

`ds.select([-1])` returns the last row, consistent with `ds[[-1]]` and with the validation layer's own semantics (its error message "Index -11 out of range for dataset of size 10" implies -10..-1 are in range). At minimum, a clean `IndexError` instead of a raw pyarrow `OverflowError`. I'll open a PR that normalizes in-range negative indices after validation.

### Environment info

- `datasets` 5.0.2.dev0 (`main` @ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOS

---
Disclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/arrow_dataset.py at _check_valid_indices_value and _select_with_indices_mapping, then run the reproduction cases from the issue. Add regression coverage for in-range negative indices, including mapped selections, and verify that valid negatives return the expected rows while out-of-range negatives still raise IndexError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.