Consider using OpenAI Evals

Nobody has claimed this yet.

OpenAI Evals is an open-source crowdsourced collection of tests designed to evaluate many newly emergent capabilities in LLMs. While it may be slightly GPT-4-centric, as tests that it can easily pass don't get merged, it still remains a valuable tool for automatically benchmarking LLMs.

While miniEvals do offer some basic questions to briefly take a look at, it can't automatically evaluate them, and isn't well focused on factual accuracy or reasoning capabilities. As open-access LLMs improve, I think it's not unreasonable to start running the same tests on them as on commercial models.

Plus, being a well-designed standard test, this may allow us to compare different open-access models against each other or even OpenAI's offerings objectively.

For reference on testing non OpenAI models with Evals, see [OpenAssistant model evals](https://github.com/tju01/oasst-automatic-model-eval).

Start by reviewing the existing miniEvals approach mentioned in the issue, then read OpenAI Evals and the referenced OpenAssistant model evals. Determine what models, capabilities, and factual-accuracy or reasoning checks should be covered, and define what an integration and comparable evaluation result would look like.

Written by the indexing model from the issue text.