allenai / allenai/olmes

potential breaking API changes from alpacal_eval?

Abierto
#13 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
395
Forks
105
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

I'm using `'preference_version': 'alpaca_eval==0.6.6'` and the current alpaca_eval_v2 in this library will throw me a key error because apparently have been change `price_per_example` to `'preference_price_per_example'`, in which the following line is affected: https://github.com/allenai/olmes/blob/38af8b61741b01faeb60e90899addd7e6fa503a0/oe_eval/tasks/oe_eval_tasks/alpaca_eval.py#L171

As well as `time_per_example`, looks like there is only `preference_price_per_example` in an annotation. Everything has been prepended with "preference". So should we pin the alpaca_eval version in this library?

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.