py-why / py-why/EconML

neg_mean_squared_error for optimizing classifier

Open
#277 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

In the notebook examples below, there is a cross validation step using neg_mean_squared_error for both first stage Regressor and classifiers. My understanding of this is that we want to minimize the mean squared error for the final model (as below).

The approach allows for arbitrary Machine Learning algorithms to be used for the two predictive tasks, while maintaining many favorable statistical properties related to the final model (e.g. small mean squared error, asymptotic normality, construction of confidence intervals). https://econml.azurewebsites.net/spec/estimation/dml.html

However, what is the implication of using a regression evaluation metric for classification cross validation?

https://github.com/microsoft/EconML/blob/master/notebooks/AutomatedML/Automated%20Machine%20Learning%20For%20EconML.ipynb
https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the cross-validation configurations in AutomatedML/Automated Machine Learning For EconML.ipynb and ForestLearners Basic Example.ipynb, focusing on classifier uses of neg_mean_squared_error. Read the linked DML specification and determine whether the examples use an appropriate classification evaluation metric; done means the behavior and its implication are clearly resolved for both notebooks.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.