juaml / juaml/julearn

[ENH]: Warn when hyperparamater tuning is hapening and now explcit scoring is used for inner CV

Open
#253 0 comments 0 reactions 1 assignee View on GitHub

@fraimondo is already working on this.

Since Mar 21, 2024.

enhancement
Dominant language
Python
Stars
33
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Which feature do you want to include?

The default in scikit-learn's SearchCV is to use the learning algorithm's score function. E.g. SVM will use accuracy. This could be a problem if the data is imbalanced. It would be better to use balanced_accuracy.

To prevent this, ideally we should warn the user if:

  1. Hyperparemeter tuning is happening
  2. Scoring is implicit

Additionally, this should include an example

How do you imagine this integrated in julearn?

In the checks, if a SearchCV is used and the scoring is None

Do you have a sample code that implements this outside of julearn?

No response

Anything else to say?

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.