SWE-Bench Verified score for ds4 + q2-imatrix

I ran a subset of 100 randomly sampled cases from total 500 of the SWE-Bench Verified benchmark using ds4 with `q2-imatrix` model as a backend for the [one-shot rig I created](https://github.com/lobanov/swe-bench-rig) (link). The results are truly impressive. Out of 100 cases (aka "instances") mini-swe-agent using ds4-serve as a backend **successfully completed 76/100 assignments**. The [headline SWE-Bench Verified score for DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash#deepseek-v4-pro-max-vs-frontier-models) is 79% for *full-precision version*. Statistically, based on this result, it's possible to say with 95% confidence that the full SWE-Bench Verified score will lie between 68.8% and 79%. It is incredibly impressive that the version of the same model having some layers quantized down to 2 bits still performs comparatively well. To put it in a perspective, [Claude 4.5 Opus scores 76.8%](https://www.swebench.com/) according to the official leaderboard. That's an excellent result and it speaks highly of the team, and of course @antirez gets the biggest credit.

**Edit**: I updated statistics-based estimate, because my original wasn't correct: this is hypergeometric distribution, not binomial.

Now, I'd love to get more accurate reading on SWE-Bench Verified on different models and architectures (Metal vs CUDA vs ROCm, anecdotally they have differerent precision), but even running these 100 instances on M5 Max took me 34.5 hours, most of which is LLM time. Because ds4 is single-threaded, running with higher parallelism is simply counterproductive.

On the other hand, if we get 25 volunteers here who can commit to an overnight run, we can have results by the following morning on a single model, so **I'm looking for volunteers**. All you would need to do is to bring your own hardware (Mac or CUDA-based) checkout the rig repo on a machine that has docker and qemo (most SWE-Bench Verified cases require x86-64 emulation), start ds4-serve, run the rig with `SWEBENCH_SLICE=N:M`, and then attach resulting csv file. I will happily aggregate the results.

Who wants to participate? Please comment below what you hardware is and when you are available, and I'll respond with an assignment of a slice.

---

**Setting up the rig**:
* Create a git worktree with specific commit: `git worktree d881f2a05e8ff6bec001315a36b794b4aa310173` (alternatively, check out commit into a detached head in existing worktree: `git checkout d881f2a05e8ff6bec001315a36b794b4aa310173`)
* Build with your architecture: `make`
* Download q2-imatrix model: `./download_model.sh q2-imatrix`
* Check hash to avoid drift: `md5sum gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf` (will take a minute); should be `b8fd44a2ec8720e5f0e481a8ba7cb390`
* Start ds4-serve: `./ds4-server --ctx 250000` (updated from `--ctx 150000` on 14th Jun). Add `--power 50` if you are concerned about heat/noise, but ok with half-speed inference. On Mac it worth running ds4-server with `caffeinate -i ./ds4-server ...` to prevent the system from going to sleep.
* Check out the rig: `gh repo clone https://github.com/lobanov/swe-bench-rig` and follow setup steps from the [README](https://github.com/lobanov/swe-bench-rig/blob/main/README.md).
* Set `LLM_BASE_URL=http://127.0.0.1:8000` in rig's `.env` (adjust hostname if running `ds4-server` elsewhere)
* Run the rig: `SWEBENCH_SLICE=N:M SWEBENCH_RUN_ID=full--- ./run.sh`. Note that docker will run linux/arm64 emulation through qemu if your CPU isn't x86-64 native, which will affect performance. It's recommended to run the rig on x86-64 native node.
* Feel free to terminate the process if needed. When you restart the rig, it'll start from the last unsubmitted instance.

**How to participate**:
1. Look for an empty row in the below matrix in the column matching your hardware: Metal for Apple, CUDA for DGX Spark or any NVIDIA, ROCm for AMD GPUs.
2. Check comments after last comment from @lobanov in case someone else has already claimed the same unallocated slice. If yes, look for another unallocated slice from the matrix
3. Post a comment to this issue saying what slice you will be running
4. Run it
5. Upon completion attach results.txt and results.csv files from your `runs` directory
6. Repeat

I will be sweeping new claims at least daily and updating the matrix. Once claimed, you have 3 days to return the results, otherwise the slice becomes unallocated again. Since this is entirely voluntary, please don't feel bad about missing the deadline, just come back when you get time to claim another slice.

| slices | Metal | CUDA (spark/generic) | ROCm |
| --- | --- | --- | --- |
| 0:20 | @antirez ✅ `[12/20]` | @ludoo ✅ `[13/20]` | @noperator by 14th Jun |
| 20:40 | @lobanov ✅ `[14/20]` | @lobanov ✅ `[11/20]` | - |
| 40:60 | @sruffilli ✅ `[15/20]` | @ludoo ✅ `[15/20]` | - |
| 60:80 | @lobanov ✅ `[16/20]` | @bbert819 ✅ `[15/20]` | - |
| 80:100 | @lobanov ✅ `[16/20]` | @bbert819 ✅ `[16/20]` | - |
| 100:120 | @daaain ✅ `[17/20]` | @bbert819 ✅ `[16/20]` | - |
| 120:140 | @lobanov ✅ `[17/20]` | @ludoo ✅ `[17/20]` | - |
| 140:160 | @lobanov ✅ `[12/20]` | @bbert819 ✅ `[14/20]` | - |
| 160:180 | @lobanov ✅ `[16/20]` | @bbert819 ✅ `[17/20]` | - |
| 180:200 | @lobanov ✅ `[18/20]` | @bbert819 ✅ `[16/20]` | - |
| 200:220 | @lobanov ✅ `[16/20]` | @bbert819 ✅ `[12/20]` | - |
| 220:240 | - | @bbert819 ✅ `[16/20]` | - |
| 240:260 | - | @bbert819 ✅ `[14/20]` | - |
| 260:280 | - | @bbert819 ✅ `[15/20]` | - |
| 280:300 | - | @bbert819 ✅ `[13/20]` | - |
| 300:320 | - | @bbert819 ✅ `[16/20]` | - |
| 320:340 | - | @bbert819 ✅ `[13/20]` | - |
| 340:360 | - | @bbert819 ✅ `[19/20]` | - |
| 360:380 | - | @bbert819 ✅ `[16/20]` | - |
| 380:400 | - | @bbert819 ✅ `[13/20]` | - |
| 400:420 | - | @bbert819 ✅ `[16/20]` | - |
| 420:440 | - | @bbert819 ✅ `[16/20]` | - |
| 440:460 | - | @bbert819 ✅ `[15/20]` | - |
| 460:480 | - | @bbert819 ✅ `[12/20]` | - |
| 480:500 | @antirez ✅ `[15/20]` | @bbert819 ✅ `[15/20]` | - |

---

Detailed results breakdown (also raw data attached as a CSV file [run100_results.csv](https://github.com/user-attachments/files/28831406/run100_results.csv))
```
Resolved: 76/100 (76.8%)

95% CI (Wilson): [67.5%, 84.0%]
Projected to full 500: 384/500 (76.8%) CI: 338–419 (67.5%–84.0%)
Run timeline
- Started: 2026-06-09 10:02:12 (UTC)
- Inference done: 2026-06-10 21:35:57 (≈34h 34m, 100/100 Submitted, 0 LimitsExceeded)
- Eval done: 2026-06-10 23:09 (≈1h 34m with --max_workers 10; would have been ~25-33h with single worker)
- Total: ~36h

=== aggregate over 100 trajs ===
total wall time : 33.40 h (120,239 s)
total LLM time : 27.07 h (97,467 s) (81.1%)
total tool time : 6.33 h (22,772 s) (18.9%)
total other : 0.00 h (0 s) (0.0%)

per-instance averages:
wall/instance : 20.0 min (max 65)
LLM/instance : 16.2 min (max 59)
tool/instance : 3.8 min (max 22)
other/instance : 0.0 min

LLM calls/instance: 64.2 (max 193)
tool calls/instance: 64.0 (max 192)

=== tokens ===
total LLM calls : 6420
total prompt tokens : 187,890,944
total cached tokens : 185,633,060 (98.8% of prompt)
total completion tokens: 2,004,667
total (prompt+completion): 189,895,611

per-instance averages:
prompt/call : 29,267
cached/call : 28,915
completion/call: 312
prompt/instance : 1,878,909
completion/instance: 20,047
```