MetaLearner with classifier as input model when target si binary

Hi,

I was wondering why scikit-uplift have better results (A LOT!) when comparing solomodel(slearner) vs econml SLearner for a X, T(binary), Y(binary) problem.

As the target is binary, I used a classifier in both metalearner models. Both used the same CatBoostClassifier() model and parameters.

When testing and experimenting very bad results for the Slearner (econml) compared to the Solomodel(scikit-uplift), I decided to change the econml Slearner estimator (overall_model) from a CatBoostClassifier() to a CatBoostRegressor().

Boom! Results were now almost the same.... WHY ?

There seems to be a problem in the effect prediction when the target is binary (Classification problems).

the ".predict" function of the given estimator is always used. Soo it means it will use by default the 0.5 threshold to classifier to a 1 or 0. It should not use the predict_proba score instead when the overall_model is a classifier ?

Start with the MetaLearner/S-Learner effect-prediction path and compare how the supplied CatBoostClassifier and CatBoostRegressor are handled. Reproduce the binary X, T, Y case using the same parameters, then inspect whether predict or predict_proba is used; done means the behavior is explained and the expected prediction path is covered by a regression test.