appwrite / appwrite/arena

Benchmark 2.0

Open
#17 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
42
Forks
6
Avg merge
4h 21m
Merged PRs (30d)
2

Description

### 🔖 Enhancement description

Details to include:

- ✅ Tokens used for each question (and average per model)
- ✅ Separate model input & output price
- ✅ Duration for each question (and average per model)
- ✅ Cost for each question (and average per model)
- ✅ TPS of model, like price of model
- ✅ total cost (sum of all)
- ✅ Remember total tool calls done for each question, and average on model
- Structure that stores old benchmarks too, not just latest

### 🎤 Pitch

More insightful benhmark

### 👀 Have you spent some time to check if this issue has been raised before?

- [x] I checked and didn't find similar issue

### 🏢 Have you read the Code of Conduct?

- [x] I have read the [Code of Conduct](https://github.com/appwrite/.github/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.