scipp / scipp/scipp

Consider dropping support for multi-dimensional bins

Open
#3,138 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
146
Forks
23
Avg merge
2d 7h
Merged PRs (30d)
7

Description

Our C++ implementation, based on MultiIndex and variable::transform supports bins with a multi-dimensional content buffer. This is rarely (if ever) used in practice by users. The current main uses are some implementation details, such as boolean indexing and indexing by integer-lists.

In #3044 we are investigating performance problems that ultimately tie into the per-bin overhead of MultiIndex. This does not just affect integer-list indexing, but any "event data" operations with very few events per bin.

A secondary problem is the extremely high complexity of MultiIndex, which potentially also affects compile times and binary sizes.

If we would consider dropping support for this (provided that the cases where this is used internally can be addressed in another manner), MultiIndex might be simplified a lot. Here is a potential solution:

  • Assume we limit ourselves to "1-D" content buffers for binned data, with stride 1.
  • variable::transform has special branches (for the purpose of optimization) for predefined stride combinations.
  • When iterating binned and dense data, this corresponds to stride-1 iteration within a bin and stride-0 for the dense operands. Therefore, MultiIndex would not need to handle this, except for loading the bin-start and bin-size. The rest could be handled by the stride-optimization branches in variable::transform.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing MultiIndex and variable::transform, then trace the boolean-indexing and integer-list cases mentioned in the issue. Check the performance investigation in #3044 and identify whether those internal uses can be handled without multi-dimensional bins. Done would require an agreed design and confirmation that existing behavior and relevant performance cases remain covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.