huggingface / huggingface/datasets
Negative percent split boundary silently anchors to the start when it rounds to 0 (`train[-1%:]` returns the whole split)
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
With the default `closest` rounding, a negative percent split boundary whose absolute value rounds to 0 loses its sign, so "relative to the end" silently becomes "relative to the start":
- `train[-1%:]` on a 30-example split returns **all 30 rows** instead of the last 1% (~0 rows, as `train[99%:]` correctly yields);
- `train[:-1%]` returns an empty slice and raises `ValueError: Instruction "train[:-1%]" corresponds to no data!` instead of the first 99% (as `train[:99%]` correctly yields).
The docs define `test[:-5%]` as "first 95% of test" ([`ReadInstruction.from_spec` docstring](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_reader.py#L554)) and use `train[-80%:]` for "last 80%" ([loading.mdx](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/docs/source/loading.mdx#L395-L402)), so the positive and negative spellings of the same boundary should agree.
Root cause: [`_rel_to_abs_instr`](https://github.com/huggingface/datasets/blob/836b82e0544cabf6474b25ade131b4d21e570373/src/datasets/arrow_reader.py#L441-L450) first rounds the boundary (`_pct_to_abs_closest(-1, 30) == int(round(-0.3)) == 0`) and then decides end-relativity from the *rounded* value (`if from_ < 0:`). When the rounded value is exactly 0, the sign is gone and the boundary anchors to the start. This triggers whenever `round(|pct| * n / 100) == 0`. The worst case is `train[-X%:]`, which silently returns 100% of the data (e.g. a "validation" split that equals the full training set).
### Steps to reproduce the bug
```python
from datasets.arrow_reader import ReadInstruction
name2len = {"train": 30}
for spec in ["train[-1%:]", "train[99%:]", "train[:-1%]", "train[:99%]"]:
abs_ = ReadInstruction.from_spec(spec).to_absolute(name2len)[0]
print(spec, "->", (abs_.from_, abs_.to))
# train[-1%:] -> (0, 30) # expected ~ (30, 30), like train[99%:]
# train[99%:] -> (30, 30)
# train[:-1%] -> (0, 0) # expected ~ (0, 30), like train[:99%]
# train[:99%] -> (0, 30)
```
Same end to end: on a 30-row CSV, `load_dataset("csv", data_files=..., split="train[-1%:]")` returns all 30 rows, and `split="train[:-1%]"` raises `ValueError: Instruction "train[:-1%]" corresponds to no data!`.
### Expected behavior
`train[-1%:]` should match `train[99%:]` (~0 rows here) and `train[:-1%]` should match `train[:99%]` (all 30 rows here): whether a boundary is end-relative should be decided by the sign written in the spec, not by the sign of the rounded absolute value. I'll open a PR with a fix.
### Environment info
- `datasets` 5.0.2.dev0 (`main` @ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOS
---
Disclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.
Contributor guide
Research direction
Start in src/datasets/arrow_reader.py, especially ReadInstruction.from_spec and _rel_to_abs_instr, then run the reproduction using the four split specifications from the issue. Verify that negative boundaries remain end-relative when rounding produces zero, and confirm the positive and negative spellings produce matching ranges.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100