np.ma.median is inefficient with DataFrames
Open
Nobody has claimed this yet.
01 - Enhancement
component: numpy.ma
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
From:
https://github.com/scikit-learn/scikit-learn/issues/19926
Using np.ma.median in a pandas DataFrame takes a lot longer than np.nanmedian. The issue seems to be about using a full sort rather than a partial argsort.
Example:
import numpy as np, pandas as pd
rng = np.random.default_rng(seed=1)
nrows = int(1e6)
ncols = 10
X = pd.DataFrame(rng.normal(size=(nrows, ncols)))
%%timeit
np.nanmedian(X, axis=0)
191 ms ± 3.98 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%%timeit
np.ma.median(X, axis=0)
949 ms ± 4.99 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
The problem does not arise when the input is a numpy array:
X2 = rng.normal(size=(nrows, ncols))
%%timeit
np.nanmedian(X2, axis=0)
188 ms ± 2.62 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%%timeit
np.ma.median(X2, axis=0)
244 ms ± 6.83 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported benchmarks for np.ma.median and np.nanmedian on a pandas DataFrame and a NumPy array. Investigate the np.ma.median path and verify whether its sorting behavior accounts for the DataFrame-specific slowdown. Done means reducing the DataFrame performance gap while preserving median results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100