Kappa computation when predicted scores are on a different scale

In a situation where predicted scores are on a completely different scale from the observed scores, kappa computation fails because the range of possible scores is too large. We saw this recently with SGDRegressor which produced scores in the range 1233372304332.22 to 1723509896207.16 when human scores were 1-5.

Few possible solutions:
(1) Do not compute kappa if there is no overlap in range between predicted and observed scores.
(2) Do not compute kappa if the range is greater than certain threshold.

Thoughts?

No se especifica ningún archivo, prueba ni punto de entrada. Empieza localizando el cálculo de kappa y reproduciendo el fallo con scores predichos alrededor de 1233372304332.22–1723509896207.16 y scores observados de 1–5. Se considerará terminada cuando se haya implementado una política acordada para rangos de scores que no se solapan o que son excesivamente grandes, y el fallo esté cubierto por una prueba.

Escrito por el modelo de indexación a partir del texto del issue.