Consider dropping support for multi-dimensional bins
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 146
- Forks
- 23
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 7
Description
Our C++ implementation, based on MultiIndex and variable::transform supports bins with a multi-dimensional content buffer. This is rarely (if ever) used in practice by users. The current main uses are some implementation details, such as boolean indexing and indexing by integer-lists.
In #3044 we are investigating performance problems that ultimately tie into the per-bin overhead of MultiIndex. This does not just affect integer-list indexing, but any "event data" operations with very few events per bin.
A secondary problem is the extremely high complexity of MultiIndex, which potentially also affects compile times and binary sizes.
If we would consider dropping support for this (provided that the cases where this is used internally can be addressed in another manner), MultiIndex might be simplified a lot. Here is a potential solution:
- Assume we limit ourselves to "1-D" content buffers for binned data, with stride 1.
variable::transformhas special branches (for the purpose of optimization) for predefined stride combinations.- When iterating binned and dense data, this corresponds to stride-1 iteration within a bin and stride-0 for the dense operands. Therefore,
MultiIndexwould not need to handle this, except for loading the bin-start and bin-size. The rest could be handled by the stride-optimization branches invariable::transform.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing MultiIndex and variable::transform, then trace the boolean-indexing and integer-list cases mentioned in the issue. Check the performance investigation in #3044 and identify whether those internal uses can be handled without multi-dimensional bins. Done would require an agreed design and confirmation that existing behavior and relevant performance cases remain covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100