py-why / py-why/EconML

Select a subset of features

Open
#409 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Sorry if I come back on this again but the examples provided until now do not work.

The problem again is that DMLIV does not authorise the usage of W and X. To replicate such behaviour, under your suggestion I am trying t use the the vector XW where XW = hstack([X[:, :(n_products)], W]) in cate.fit(Y, T[:,0], X=XW, Z=Z, cache_values=True, inference='auto') and afterwards I would like to keep just X[:, :(n_products)] in the final model s that I am able at the end t output:

X_test = np.vstack([np.eye(n_products)])
dmliv_cate = cate.effect(X_test)
dmliv_cate

which am unable to do until now. The dimensions are: (517, 76) for XW, (517,11) fr X[:, :(n_products)], (11,11) for X_test, (517,2) for T[:,0] and (517,1) for Y.
I am currently trying with:

cate = DMLIV(model_Y_X= model_Y_X(), model_T_X = model_T_X(), model_T_XZ= model_T_XZ(), model_final=dmliv_model_effect(),
             featurizer=ftur,cv=6, random_state=123)

where ftur = sklearn.compose.ColumnTransformer([('passthrough','passthrough',[0,1,2,3,4,5,6,7,8,9,10])]) but when trying:

X_test = np.vstack([np.eye(n_products)])
dmliv_cate = cate.effect(X_test)
dmliv_cate

the dimensionality error appears (pointing to the fact that features fitted have dimension different from test ones, which I did not expect since ColuumnTransformer should select the subset f first 10 columns)

EDIT: I tried as featurizer:

sklearn.compose.ColumnTransformer([('passthrough','passthrough',[0,1,2,3,4,5,6,7,8,9,10])])

but I end up having (517) identica coefficients when doing:

dmliv_cate = cate.effect(XW)
dmliv_cate

I really cannot understand the reason why when I was not controlling for endogeneity the whole process worked fine. I just would like to reproduce something as easy as:

est.fit(Y, T, X[:, :(n_products)], W, inference='debiasedlasso')

but with est being DMLIV and then using as test a (11,11) identity matrix, which worked very well when using SparseLinearDMLCateEstimator() which however cannot take into account the endogeneity given by the reverse causality of price vs quantity (indeed also in the documentation I cannot understand why it is not taken into account such problem)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DMLIV fit and effect calls in the issue, including the ColumnTransformer featurizer and the XW versus X_test dimensions. Compare this behavior with SparseLinearDMLCateEstimator and the documented DMLIV interface; done means establishing whether selecting a feature subset for fitting and effect evaluation is supported or identifying the required clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.