Improve performance of `tabmat.from_pandas` for sparse columns
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 140
- Forks
- 10
- Avg merge
- 18h 17m
- Merged PRs (30d)
- 2
Description
Instead of
we should consider
if isinstance(coldata.dtype, pd.SparseDtype) and coldata.sparse.fill_value == 0:
sparse_density = coldata.sparse.density
else:
sparse_density = (coldata != 0).mean()
if sparse_density <= sparse_threshold:
...
pandas.Series.sparse.density operates in constant time (e.g., 3e-5s on my machine) while pandas.Series.ne and pandas.Series.mean are linear in the amount of non-zero entries for sparse columns (e.g., 3e-3s on my machine for one million rows with density 0.1).
The proposed changes reduce the time to convert a pandas.DataFrame with 130 sparse columns (and a few categoricals) and one million rows from 0.5s to 0.15s.
Edit: It's possible for pandas.Series.sparse.sp_values to contain zeros (e.g., when multiplying two sparse arrays, their product's sp_index is the union rather than the intersection of the factors' sp_index). In particular, coldata.sparse.density and (coldata != 0).mean() are not equivalent in those cases. Because scipy.sparse.coo_matrix uses pandas.Series.sparse.sp_values though, coldata.sparse.density appears to be the more sensible solution. Consider the below example with coldata.sparse.density = 1 and (coldata != 0).mean() = 0 which results in a "sparse" matrix with 2000 zero entries with the current tabmat version.
import pandas as pd
import tabmat as tm
df = (
pd.DataFrame(
{
"col_a": [1, 0] * 1000,
"col_b": [0, 1] * 1000,
}
)
.astype(pd.SparseDtype(fill_value=0))
.assign(col_c=lambda x: x["col_a"] * x["col_b"])
)
tm.from_pandas(df.filter(["col_c"]))._array.shape
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/tabmat/constructor.py around line 158 and inspect how from_pandas calculates sparse density. Compare pandas sparse density with the existing calculation, including the provided col_c example, then verify that conversion chooses the appropriate representation and preserves the reported matrix shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100