EducationalTestingService / EducationalTestingService/rsmtool

Kappa computation when predicted scores are on a different scale

Aperta
#342 10 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Python
Stelle
71
Fork
21
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

In a situation where predicted scores are on a completely different scale from the observed scores, kappa computation fails because the range of possible scores is too large. We saw this recently with SGDRegressor which produced scores in the range 1233372304332.22 to 1723509896207.16 when human scores were 1-5.

Few possible solutions:
(1) Do not compute kappa if there is no overlap in range between predicted and observed scores.
(2) Do not compute kappa if the range is greater than certain threshold.

Thoughts?

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.