Layr-Labs / Layr-Labs/d-inference

[bug] 32 GB M1 Max already runs Qwen3.6-35B-A3B at ~42 tok/s / 21–23 GB — provider admission still demands 30.3 GB

Open
#724 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
522
Forks
113
Avg merge
17h 26m
Merged PRs (30d)
111

Description

### Component

provider-swift (Swift CLI)

### What happened?

Dedicated Mac Studio M1 Max 32 GB (Mac13,1, 24 GPU cores), macOS 26.6.2, Darkbloom 0.8.11, SIP on, MDM hardware trust, APNs console session (auto-login).
Box stripped (no browser / LM Studio / extra GPU clients).
Catalog lists qwen3.6-35b-a3b-vl-mtp-mxfp8 as eligible (≥ 32 GB RAM, fp4/MXFP8, 21.3 GB). Provider will not load it. Dashboard stays hardware-trusted / ONLINE with 0 tokens.
Empty-box `darkbloom doctor` (gpt-oss unloaded): - Inference memory: 28 GB (UnifiedMemoryCap 0.90) - metal gpu: Apple M1 Max, 24 GB working set - 23.3 GB usable - gpt-oss-20b needs ~20.0 GB → PASS (goes warm) 35B load need on 0.8.11: EST MEM 23.8 + activation 5.5 + min KV 1.0 = 30.3 GB hard cap at fraction 1.0 = min(32, 32−2) = 30.0 GB
Typical error: Insufficient memory (X GB free, need 30.3 GB) and all loaded models are actively serving (that last clause is boilerplate even with zero models loaded)
Same chip already served the same model class outside Darkbloom:
MTPLX 2.9.1, Youssofal/Qwen3.6-35B-A3B-MTPLX-Optimized-Speed-FP16, sustained, MTP D1, thinking off, 16K ctx. Peak RAM 20.6–23.3 GB, no jetsam. Decode (raw /v1, no Hermes tools): - Short chat: 42.0 / 41.8 tok/s - Short AR (no MTP): 43.2 tok/s - ~512 ctx: 42.1 tok/s - ~2K ctx: 39.2 tok/s - ~8K ctx: 36.9 tok/s - 256 out D1/AR: 43.2 / 43.4 tok/s (D2 slower: 37.6)
So the 21 GB 35B-A3B MoE (~3B active) already fits and runs at ~37–43 tok/s inside the 23.3 GB Darkbloom called “usable”.
The extra 5.5+1.0 in the gate is what pushes need to 30.3 and refuses the catalog file.

DARKBLOOM_ACTIVATION_RESERVE_GB below 5.5 is clamped. This is not “close more apps.”

### Expected behavior

Either:
1) A dedicated 32 GB provider (max_model_slots = 1) can load qwen3.6-35b-a3b-vl-mtp-mxfp8, since this chip already holds a 21 GB 35B-A3B MLX pack at 20.6–23.3 GB peak; or
2) Catalog min_ram_gb for that id is 48, matching the 30.3 GB gate, so 32 GB nodes are not shown as eligible / ONLINE / EARNING with 0 tokens.

Bare minimum if you keep the model at ≥32:
- Do not use Metal recommendedMaxWorkingSetSize (24 GB here) as the load-gate “usable”; use UnifiedMemoryCap / live free (28 GB at 0.90, 30 GB at 1.0) for max_model_slots = 1.
- Single-slot activation 3 GiB (pre-0.8 floor / B=1), not flat 5.5 (B=8). Then need = 23.8+3+1 = 27.8 < 28.8 cap.
Optional: EST MEM = catalog size_gb 21.3 (matches measured peak).

Smallest single change: gate on measured/weight size (~21–23 GB) instead of EST+5.5+1.

### Steps to reproduce

1. Apple Silicon Mac with 32 GB unified memory (here: M1 Max Studio Mac13,1).
2. Install current provider (0.8.11): curl -fsSL https://api.darkbloom.dev/install.sh | bash
3. darkbloom login && darkbloom enroll (hardware trust). Keep a console Aqua session (auto-login). Strip other GPU/RAM hogs.
4. darkbloom models download qwen3.6-35b-a3b-vl-mtp-mxfp8
5. darkbloom start --model qwen3.6-35b-a3b-vl-mtp-mxfp8
6. darkbloom status && darkbloom doctor

Observe: trust hardware / online; Warm models: none; last load error need 30.3 GB; doctor FAIL model fits in RAM; tokens 0.

Contrast (optional): same machine, MLX/MTPLX Qwen3.6-35B-A3B ~21 GB pack loads at 20.6–23.3 GB peak and decodes ~37–43 tok/s.

### Environment

- provider: darkbloom 0.8.11
- macOS: 26.6.2 (Tahoe)
- hardware: Mac Studio Mac13,1, Apple M1 Max, 32 GB unified, 24 GPU cores
- SIP: enabled; MDM: enrolled; trust: hardware
- config: memory_reserve_gb = 1; max_model_slots = 1
- empty usable: 23.3 GB
- Metal working set: 24 GB
- UnifiedMemoryCap (0.90): 28 GB
- 35B MLX peak (this chip): 20.6–23.3 GB @ 37–43 tok/s
- Darkbloom 35B need: 30.3 GB → refused
- gpt-oss-20b: loads (need ~20.0 GB)

### Logs

```shell
darkbloom doctor (empty / gpt-oss unloaded):
[FAIL] model fits in RAM — qwen3.6-35b-a3b-vl-mtp-mxfp8 needs ~30.3 GB but only ~23.3 GB is usable
↳ it will show online but every request fails to load

darkbloom status (when pinned to 35B):
Trust: hardware / online
Warm models: none loaded
Last model-load error: qwen3.6-35b-a3b-vl-mtp-mxfp8: Insufficient memory (X GB free, need 30.3 GB) and all loaded models are actively serving
Requests served: 0 | tokens: 0

Same host, MTPLX 35B-A3B (not Darkbloom): peak 20.6–23.3 GB, 42.0 tok/s short decode.
```

### Confirmations

- [x] I searched existing issues and this isn't a duplicate

Contributor guide

Open the contributing guide

Research direction

Start at the provider-swift model-load admission path and compare its memory calculation with the values reported by darkbloom doctor on a 32 GB Apple Silicon host. Reproduce with darkbloom models download, darkbloom start, darkbloom status, and darkbloom doctor; done means the single-slot node can load the model or its catalog eligibility matches the enforced memory requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
ai-infra-agents, cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.