"Likelihood" based functions are not implemented for API-based models
Abierto
- Lenguaje dominante
- Python
- Estrellas
- 395
- Forks
- 105
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
I cannot run the eval with API-based models using model-type LiteLLM. I get the error: "NotImplementedError: loglikelihood_verbose not implemented for LiteLLM". I see that these functions are not implemented for litellm in https://github.com/allenai/olmes/blob/38af8b61741b01faeb60e90899addd7e6fa503a0/oe_eval/models/litellm.py. Are these functions going to be added later or are they not added for a reason?
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.