NVIDIA / NVIDIA/cudf

[Story]: Reworking of `iloc` and `loc` indexing

Open
#12,793 7 comments 4 reactions 0 assignees View on GitHub
improvement Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Status quo

Indexing of dataframes and series happens through six user-facing routes:

- `DataFrame.__setitem__`/`DataFrame.__getitem__`
- `DataFrame.iloc.__setitem__`/`DataFrame.iloc.__getitem__`
- `DataFrame.loc.__setitem__`/`DataFrame.loc.__getitem__`
- `Series.__setitem__`/`Series.__getitem__`
- `Series.iloc.__setitem__`/`Series.iloc.__getitem__`
- `Series.loc.__setitem__`/`Series.loc.__getitem__`

These all have slightly different semantics (to match pandas behaviour), but there is still quite a lot of (possibly unnecessary) code duplication and a number of bugs around indexing. Many of these look to be because the business logic of handling slicing/gather-by-mask/indexing is intertwined with error handling and determining exactly what to slice. There's also logic effectively repeated between the loc and iloc versions in both cases.

It would be nice if the number of different paths into indexing was reduced, perhaps it is a pipe dream to share between Series and DataFrame (since a DataFrame is not just a collection of Series), but it feels like it should be possible to share more between iloc/loc/__setgetitem__.

Related issues:

### `iloc` bugs
- [ ] #12748
- [ ] #13013
- [ ] #13015
- [ ] #13265
- [ ] #13266
- [ ] #13267
- [ ] #13515
- [ ] #13293

### Index bugs
- [ ] #12954

### `loc` bugs
- [ ] #7448
- [ ] #8585
- [ ] #8693
- [ ] #11298
- [ ] #11944
- [ ] #12259
- [ ] #12286
- [ ] #12504
- [ ] #12505
- [ ] #12801
- [ ] #12833
- [ ] #13014
- [ ] #13015
- [ ] #13031
- [ ] #13268
- [ ] #13269
- [ ] #13270
- [ ] #13379
- [ ] https://github.com/rapidsai/cudf/issues/13653
- [ ] https://github.com/rapidsai/cudf/issues/13658
- [ ] https://github.com/rapidsai/cudf/issues/13652

### `at`/`iat` bugs

- [x] https://github.com/rapidsai/cudf/issues/20561

### Views vs. copies
- [ ] #7374
- [ ] #11085
- [x] #11990

### Other (mostly dtype-related)
- [ ] #2684
- [ ] #8184
- [ ] #11477
- [ ] #12039
- [ ] #13532

As we can see from this classification, `loc`-based indexing is definitely the harder nut to crack. The edge-cases that provoke most of the issues are cases where the values used in the indexing are _not_ in the index.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.