py-why / py-why/EconML

Sparse matrix features for DMLIV

Open
#508 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

I have been trying to use DMLIV for a problem where we want to estimate fixed effects of many features, in our case about 75000 of them. We have about 5.5 million observations. I have to use sparse matrices for the features to avoid running out of memory (256GB). What is the proper way to use DMLIV with sparse matrix features?

To illustrate what I have tried, the data looks something like

n = 5493141
p = 75000
y = np.random.poisson(size=(n, 1))
t = np.random.binomial(1, .001, size=(n, 1))
z = np.random.binomial(1, .001, size=(n, 1))
z_spmat = scipy.sparse.csr_matrix(z)

density = 0.00000001
size = int(n * p * density)

rows = np.random.randint(0, 2, size=size)
cols = np.random.randint(0, 2, size=size)
data = np.random.rand(size)

x_spmat = scipy.sparse.csr_matrix((data, (rows, cols)), shape=(n, p))
x_df = pd.DataFrame.sparse.from_spmatrix(x)

I am using SGD classifiers/regressors for first stage models, which I am able to fit to my data outside of econml:

from sklearn.linear_model import SGDClassifier, SGDRegressor

occ_sgd3 = DMLIV(
  model_Y_X = SGDRegressor(),
  model_T_X = SGDClassifier(loss='log'),
  model_T_XZ = SGDClassifier(loss='log'),
  model_final = SGDRegressor(),
)

When I pass in the features x as a sparse matrix:

occ_sgd3.fit(Y=y, T=t, Z=z, X=x_spmat)

I get the error IndexError: tuple index out of range.

When I pass them in as a pandas data frame with sparse columns

occ_sgd3.fit(Y=y, T=t, Z=z, X=x_df)

I get ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 1 dimension(s).

After digging around a little I tried making z a sparse matrix, since it seems to get concatenated with x when fitting, and trying

occ_sgd3.fit(Y=y, T=t, Z=z_spmat, X=x_df)

gives ValueError: Found input variables with inconsistent numbers of samples: [2, 5493141].

Finally I tried to avoid cross fitting by setting cv=1; I seemed to get further and actually estimated the first stage models, but eventually get the error TypeError: 'coo_matrix' object is not subscriptable. It looks like there's code in utilities.py that handles sparse matrix input so I suspect my input formats are not quite right.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DMLIV.fit entry point and the sparse-input handling in utilities.py, then reproduce the reported fits with x_spmat, x_df, z_spmat, and cv=1. Trace the shape and indexing failures; done means establishing a supported sparse input format or a clearly scoped correction for these errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python, scikit-learn
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.