[ENH]: Warn when hyperparamater tuning is hapening and now explcit scoring is used for inner CV
Open
@fraimondo is already working on this.
Since Mar 21, 2024.
enhancement
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Which feature do you want to include?
The default in scikit-learn's SearchCV is to use the learning algorithm's score function. E.g. SVM will use accuracy. This could be a problem if the data is imbalanced. It would be better to use balanced_accuracy.
To prevent this, ideally we should warn the user if:
- Hyperparemeter tuning is happening
- Scoring is implicit
Additionally, this should include an example
How do you imagine this integrated in julearn?
In the checks, if a SearchCV is used and the scoring is None
Do you have a sample code that implements this outside of julearn?
No response
Anything else to say?
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.