neg_mean_squared_error for optimizing classifier
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
In the notebook examples below, there is a cross validation step using neg_mean_squared_error for both first stage Regressor and classifiers. My understanding of this is that we want to minimize the mean squared error for the final model (as below).
The approach allows for arbitrary Machine Learning algorithms to be used for the two predictive tasks, while maintaining many favorable statistical properties related to the final model (e.g. small mean squared error, asymptotic normality, construction of confidence intervals). https://econml.azurewebsites.net/spec/estimation/dml.html
However, what is the implication of using a regression evaluation metric for classification cross validation?
https://github.com/microsoft/EconML/blob/master/notebooks/AutomatedML/Automated%20Machine%20Learning%20For%20EconML.ipynb
https://github.com/microsoft/EconML/blob/master/notebooks/ForestLearners%20Basic%20Example.ipynb
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the cross-validation configurations in AutomatedML/Automated Machine Learning For EconML.ipynb and ForestLearners Basic Example.ipynb, focusing on classifier uses of neg_mean_squared_error. Read the linked DML specification and determine whether the examples use an appropriate classification evaluation metric; done means the behavior and its implication are clearly resolved for both notebooks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100