Quantco / Quantco/tabmat

Improve performance of `tabmat.from_pandas` for sparse columns

Open
#378 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Python
Stars
140
Forks
10
Avg merge
18h 17m
Merged PRs (30d)
2

Description

Instead of

https://github.com/Quantco/tabmat/blob/c0c8626b0b35737e34742c422f6adc41d0b8001f/src/tabmat/constructor.py#L158

we should consider

if isinstance(coldata.dtype, pd.SparseDtype) and coldata.sparse.fill_value == 0:
    sparse_density = coldata.sparse.density
else:
    sparse_density = (coldata != 0).mean()

if sparse_density <= sparse_threshold:
    ...

pandas.Series.sparse.density operates in constant time (e.g., 3e-5s on my machine) while pandas.Series.ne and pandas.Series.mean are linear in the amount of non-zero entries for sparse columns (e.g., 3e-3s on my machine for one million rows with density 0.1).

The proposed changes reduce the time to convert a pandas.DataFrame with 130 sparse columns (and a few categoricals) and one million rows from 0.5s to 0.15s.

Edit: It's possible for pandas.Series.sparse.sp_values to contain zeros (e.g., when multiplying two sparse arrays, their product's sp_index is the union rather than the intersection of the factors' sp_index). In particular, coldata.sparse.density and (coldata != 0).mean() are not equivalent in those cases. Because scipy.sparse.coo_matrix uses pandas.Series.sparse.sp_values though, coldata.sparse.density appears to be the more sensible solution. Consider the below example with coldata.sparse.density = 1 and (coldata != 0).mean() = 0 which results in a "sparse" matrix with 2000 zero entries with the current tabmat version.

import pandas as pd
import tabmat as tm

df = (
    pd.DataFrame(
        {
            "col_a": [1, 0] * 1000,
            "col_b": [0, 1] * 1000,
        }
    )
    .astype(pd.SparseDtype(fill_value=0))
    .assign(col_c=lambda x: x["col_a"] * x["col_b"])
)
tm.from_pandas(df.filter(["col_c"]))._array.shape

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/tabmat/constructor.py around line 158 and inspect how from_pandas calculates sparse density. Compare pandas sparse density with the existing calculation, including the provided col_c example, then verify that conversion chooses the appropriate representation and preserves the reported matrix shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.