Select a subset of features
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Sorry if I come back on this again but the examples provided until now do not work.
The problem again is that DMLIV does not authorise the usage of W and X. To replicate such behaviour, under your suggestion I am trying t use the the vector XW where XW = hstack([X[:, :(n_products)], W]) in cate.fit(Y, T[:,0], X=XW, Z=Z, cache_values=True, inference='auto') and afterwards I would like to keep just X[:, :(n_products)] in the final model s that I am able at the end t output:
X_test = np.vstack([np.eye(n_products)])
dmliv_cate = cate.effect(X_test)
dmliv_cate
which am unable to do until now. The dimensions are: (517, 76) for XW, (517,11) fr X[:, :(n_products)], (11,11) for X_test, (517,2) for T[:,0] and (517,1) for Y.
I am currently trying with:
cate = DMLIV(model_Y_X= model_Y_X(), model_T_X = model_T_X(), model_T_XZ= model_T_XZ(), model_final=dmliv_model_effect(),
featurizer=ftur,cv=6, random_state=123)
where ftur = sklearn.compose.ColumnTransformer([('passthrough','passthrough',[0,1,2,3,4,5,6,7,8,9,10])]) but when trying:
X_test = np.vstack([np.eye(n_products)])
dmliv_cate = cate.effect(X_test)
dmliv_cate
the dimensionality error appears (pointing to the fact that features fitted have dimension different from test ones, which I did not expect since ColuumnTransformer should select the subset f first 10 columns)
EDIT: I tried as featurizer:
sklearn.compose.ColumnTransformer([('passthrough','passthrough',[0,1,2,3,4,5,6,7,8,9,10])])
but I end up having (517) identica coefficients when doing:
dmliv_cate = cate.effect(XW)
dmliv_cate
I really cannot understand the reason why when I was not controlling for endogeneity the whole process worked fine. I just would like to reproduce something as easy as:
est.fit(Y, T, X[:, :(n_products)], W, inference='debiasedlasso')
but with est being DMLIV and then using as test a (11,11) identity matrix, which worked very well when using SparseLinearDMLCateEstimator() which however cannot take into account the endogeneity given by the reverse causality of price vs quantity (indeed also in the documentation I cannot understand why it is not taken into account such problem)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the DMLIV fit and effect calls in the issue, including the ColumnTransformer featurizer and the XW versus X_test dimensions. Compare this behavior with SparseLinearDMLCateEstimator and the documented DMLIV interface; done means establishing whether selecting a feature subset for fitting and effect evaluation is supported or identifying the required clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100