EducationalTestingService / EducationalTestingService/rsmtool

Kappa computation when predicted scores are on a different scale

Open
#342 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
71
Forks
21
PR merge metrics
No merged PRs in 30d

Description

In a situation where predicted scores are on a completely different scale from the observed scores, kappa computation fails because the range of possible scores is too large. We saw this recently with SGDRegressor which produced scores in the range 1233372304332.22 to 1723509896207.16 when human scores were 1-5.

Few possible solutions:
(1) Do not compute kappa if there is no overlap in range between predicted and observed scores.
(2) Do not compute kappa if the range is greater than certain threshold.

Thoughts?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.