numpy / numpy/numpy

structured arrays slow to mask

Open
#17,850 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

I have 5 arrays of the same data type and the same length n (can be quite large). I need to regularly mask all of those array with the same mask, and I thought about what would be the fastest way. Fitting the data into one (n, 5) array works, and masking is equally fast with a (5, n) array.

I'd love to use a structured array so I can properly name the data in the rest of the code (e.g., arr["density"] instead of arr[:, 3]). Unfortunately, structured arrays mask way slower than the other two options:

out

Code to reproduce the plot:

import numpy
import perfplot
import random


def setup(n):
    a = numpy.random.rand(n, 5)
    b = numpy.ascontiguousarray(a.T)
    c = a.copy()
    c.dtype = [(f"col{k}", c.dtype) for k in range(a.shape[1])]

    idx = random.sample(range(0, n), 25)
    mask = numpy.ones(n, dtype=bool)
    mask[idx] = False
    return a, b, c, mask


def mask_rows(data):
    a, _, _, mask = data
    return a[mask]


def mask_named(data):
    _, _, c, mask = data
    return c[mask]


def mask_cols(data):
    _, b, _, mask = data
    return b[:, mask]


perfplot.show(
    setup=setup,
    kernels=[mask_rows, mask_cols, mask_named],
    n_range=[2 ** k for k in range(5, 20)],
    equality_check=None,
)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python reproduction and comparing boolean masking for the two-dimensional arrays and the structured array. Investigate the structured-array masking path responsible for the difference. Done means structured-array masking is no longer disproportionately slower, with the reproduction confirming the improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.