structured arrays slow to mask
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
I have 5 arrays of the same data type and the same length n (can be quite large). I need to regularly mask all of those array with the same mask, and I thought about what would be the fastest way. Fitting the data into one (n, 5) array works, and masking is equally fast with a (5, n) array.
I'd love to use a structured array so I can properly name the data in the rest of the code (e.g., arr["density"] instead of arr[:, 3]). Unfortunately, structured arrays mask way slower than the other two options:

Code to reproduce the plot:
import numpy
import perfplot
import random
def setup(n):
a = numpy.random.rand(n, 5)
b = numpy.ascontiguousarray(a.T)
c = a.copy()
c.dtype = [(f"col{k}", c.dtype) for k in range(a.shape[1])]
idx = random.sample(range(0, n), 25)
mask = numpy.ones(n, dtype=bool)
mask[idx] = False
return a, b, c, mask
def mask_rows(data):
a, _, _, mask = data
return a[mask]
def mask_named(data):
_, _, c, mask = data
return c[mask]
def mask_cols(data):
_, b, _, mask = data
return b[:, mask]
perfplot.show(
setup=setup,
kernels=[mask_rows, mask_cols, mask_named],
n_range=[2 ** k for k in range(5, 20)],
equality_check=None,
)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python reproduction and comparing boolean masking for the two-dimensional arrays and the structured array. Investigate the structured-array masking path responsible for the difference. Done means structured-array masking is no longer disproportionately slower, with the reproduction confirming the improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100