scikit-learn / scikit-learn/scikit-learn

cohen's kappa can't handle single label cases

Open
#9,624 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module:metrics
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

cohen_kappa_score assumes that the data is going to have multiple labels, although most time's that is the case for training a classifier that may not always be the case for testing, like for example if you are just evaluating a classifier over many datasets and one of them only has one class data. Sure this evaluation is kind of useless but this will happen when running a classifier through unknown datasets where you are trying to estimate classification performance.

Here how the problem could be reproduced:

from metrics import cohen_kappa_score
cohen_kappa_score([0,0],[0,0])

The above results in a nan, when in reality it should throw an error, or return a 1 with a warning. The one is because after all the classifier we are evaluating in this case predicted perfectly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The entry point is cohen_kappa_score; first run the supplied two-line reproduction and inspect its implementation and related metric tests. Resolve the expected single-label behavior—an error or a warning with a result of 1—and add regression coverage for the chosen outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.