huggingface / huggingface/datasets

Negative percent split boundary silently anchors to the start when it rounds to 0 (`train[-1%:]` returns the whole split)

Open Beginner friendly
#8,473 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

With the default `closest` rounding, a negative percent split boundary whose absolute value rounds to 0 loses its sign, so "relative to the end" silently becomes "relative to the start":

- `train[-1%:]` on a 30-example split returns **all 30 rows** instead of the last 1% (~0 rows, as `train[99%:]` correctly yields);
- `train[:-1%]` returns an empty slice and raises `ValueError: Instruction "train[:-1%]" corresponds to no data!` instead of the first 99% (as `train[:99%]` correctly yields).

The docs define `test[:-5%]` as "first 95% of test" ([`ReadInstruction.from_spec` docstring](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_reader.py#L554)) and use `train[-80%:]` for "last 80%" ([loading.mdx](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/docs/source/loading.mdx#L395-L402)), so the positive and negative spellings of the same boundary should agree.

Root cause: [`_rel_to_abs_instr`](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_reader.py#L441-L450) first rounds the boundary (`_pct_to_abs_closest(-1, 30) == int(round(-0.3)) == 0`) and then decides end-relativity from the *rounded* value (`if from_ < 0:`). When the rounded value is exactly 0, the sign is gone and the boundary anchors to the start. This triggers whenever `round(|pct| * n / 100) == 0`. The worst case is `train[-X%:]`, which silently returns 100% of the data (e.g. a "validation" split that equals the full training set).

### Steps to reproduce the bug

```python
from datasets.arrow_reader import ReadInstruction

name2len = {"train": 30}
for spec in ["train[-1%:]", "train[99%:]", "train[:-1%]", "train[:99%]"]:
abs_ = ReadInstruction.from_spec(spec).to_absolute(name2len)[0]
print(spec, "->", (abs_.from_, abs_.to))
# train[-1%:] -> (0, 30) # expected ~ (30, 30), like train[99%:]
# train[99%:] -> (30, 30)
# train[:-1%] -> (0, 0) # expected ~ (0, 30), like train[:99%]
# train[:99%] -> (0, 30)
```

Same end to end: on a 30-row CSV, `load_dataset("csv", data_files=..., split="train[-1%:]")` returns all 30 rows, and `split="train[:-1%]"` raises `ValueError: Instruction "train[:-1%]" corresponds to no data!`.

### Expected behavior

`train[-1%:]` should match `train[99%:]` (~0 rows here) and `train[:-1%]` should match `train[:99%]` (all 30 rows here): whether a boundary is end-relative should be decided by the sign written in the spec, not by the sign of the rounded absolute value. I'll open a PR with a fix.

### Environment info

- `datasets` 5.0.2.dev0 (`main` @ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOS

---
Disclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/arrow_reader.py, especially ReadInstruction.from_spec and _rel_to_abs_instr, then run the reproduction using the four split specifications from the issue. Verify that negative boundaries remain end-relative when rounding produces zero, and confirm the positive and negative spellings produce matching ranges.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.