What's the recommended way to test local models?
Open
- Dominant language
- Python
- Stars
- 395
- Forks
- 105
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'm wondering what's the recommended way to run Olmes against new models running locally. I can spin up a vLLM/SGLang instance to serve my model, but it seems like the only way to configure a new model is by adding it to [models.py](https://github.com/allenai/olmes/blob/main/oe_eval/configs/models.py). Is that correct or is there a more seamless approach?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.