Periodic OpenCode Zen Model & Effort-Level Benchmarking

Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

I’m a user of OpenCode and the OpenCode Zen API, and I have a suggestion that I think would be extremely valuable to the community.

Would OpenCode consider periodically benchmarking all model/effort-level combinations available through the Zen API and publishing the results on the OpenCode website?

Zen is particularly well positioned to provide this because you control the model/provider configurations and already test and benchmark the models you make available. At the moment, however, it’s difficult for users to answer a very practical question:

Which Zen model and reasoning-effort level gives me the best coding performance for the money?

Artificial Analysis provides excellent coding-agent benchmarks, but its coverage of OpenCode/Zen model-effort combinations is relatively limited. OpenCode could potentially fill this gap with an authoritative benchmark specifically representing the Zen experience.

Suggested benchmark

For every supported Zen model, test each applicable effort/reasoning level using the same standardized set of coding-agent tasks, for example:

• Repository understanding
• Feature implementation
• Bug fixing
• Refactoring
• Test creation/fixing
• Multi-file architectural changes

For each model/effort combination, publish:

• Overall benchmark score
• Task success rate
• Tests passed
• Average/median completion time
• Input/output/reasoning tokens
• Cost per task
• Failure rate
• Human intervention, if applicable

The resulting table could look something like:

Model	Effort	Score	Success	Time	Tokens	Cost/task
GPT-5.6 Luna	low	…	…	…	…	…
GPT-5.6 Luna	medium	…	…	…	…	…
GPT-5.6 Luna	high	…	…	…	…	…
GPT-5.6 Luna	max	…	…	…	…	…
Gemini 3.8 Flash	high	…	…	…	…	…
Muse Spark	xhigh	…	…	…	…	…

I would also strongly recommend publishing a Pareto frontier of quality vs. cost, since that would make the data considerably more useful for selecting models.

Periodic re-testing

Ideally, this would be repeated whenever the Zen catalog changes materially—perhaps monthly or quarterly—with historical results retained.

That would allow users to see not only which models perform best, but also how the economics and performance of the Zen ecosystem are changing over time.

Why I think this would be particularly useful

One of the most interesting things about Zen is that it gives users access to multiple models through a relatively consistent coding-agent environment. That makes it possible to compare the actual model + OpenCode harness + effort configuration, rather than relying on generic model benchmarks.

It would also help answer questions such as:

• Is GPT-5.6 Luna high actually worth the additional cost over medium?
• Does increasing reasoning effort materially improve task completion?
• Which inexpensive models are genuinely competitive?
• Where is the quality/cost sweet spot?
• Which models are best for repository-scale engineering versus simple coding tasks?
• Does a model’s performance change significantly when operated through OpenCode compared with another agent harness?

A public benchmark like this could become one of the strongest differentiators of OpenCode Zen.

Even a relatively small standardized benchmark would be useful. The computational/API cost should be quite manageable if the benchmark consists of a modest number of representative tasks, and the results could be published alongside the existing Zen model catalog.

I’d be very interested in seeing something like an “OpenCode Zen Coding Agent Benchmark” become a regularly updated public resource.

Thanks for building OpenCode and Zen. It’s a very compelling approach, and I think transparent model/effort benchmarking would make it even more useful for serious users trying to choose the right configuration.

Best regards,

Michael