apache / apache/apisix

onthebench.ai benchmarks APISIX: a review of our setup for fairness

Open
#13,740 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

Hi, I run [onthebench.ai](https://onthebench.ai), an open benchmark that measures LLM-gateway overhead (latency, throughput, memory, streaming, protocol translation) on neutral hardware. **APISIX** is one of the gateways on the board, and I want to make sure I'm testing it fairly.

How it works, so there are no surprises:
- Every gateway runs on the **same rig, same mock upstream, same load, same CPU pinning**, no per-gateway special-casing.
- Each gateway is defined by a single file, [`gateways/apisix/gateway.sh`](https://github.com/GetBusbar/benchmarking/blob/main/gateways/apisix/gateway.sh), which declares how to build, launch, and probe it. That file is the whole story of how I configured yours.
- Every number regenerates from committed JSON; the [method is documented here](https://onthebench.ai/gateways/method) and the whole thing is open source and re-runnable.

My guiding rule is that **a failure is my bug until proven the gateway's**. If a cell shows red or a number looks off, I'd rather find out I configured **APISIX** wrong than publish something unfair. So two asks, both optional:
1. **Look at your results** ([your page](https://onthebench.ai/gateways?gw=apisix)) and your `gateway.sh`. If I've mis-set a flag, a version, an endpoint, or declared a capability you don't claim, tell me or open a PR, you're welcome to own your own `gateway.sh`.
2. If the setup looks fair, a thumbs-up is genuinely useful too.

Full disclosure: this is built and operated by the Busbar team, and busbar is also one of the entrants. Same harness for everyone, fully open, so you can verify exactly what I'm running.

Thanks for building **APISIX**.

Contributor guide

Open the contributing guide

Research direction

Review the external benchmark results and the linked gateways/apisix/gateway.sh, then compare the setup with the documented method at onthebench.ai/gateways/method. Done means identifying and reporting any incorrect flag, version, endpoint, or capability declaration, or confirming that the configuration is fair.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, shell
Domain
backend-api-design, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.