RFC: Opportunistic batch inference on spare provider capacity

### Component

coordinator (Go)

### Problem / motivation

Could Darkbloom use spare capacity on already-loaded providers to serve delay-tolerant inference at a lower price?

Historical routing and base-rewards discussions describe underused capacity. Those snapshots motivate this proposal; they are not measurements of current fleet utilization. Discounted batch traffic could provide additional paid work when demand exists, alongside existing provider incentives.

I have built a prototype in [rajagurunath/d-inference](https://github.com/rajagurunath/d-inference/tree/2238cf273). It adapts Tidal's gateway approach—capacity-aware admission, AIMD and deadline-based ordering—to the Go coordinator. The contribution is this integration, rather than a claim that these scheduling techniques are new.

The proposed defaults are a **24-hour completion window** and a **0.5 multiplier on resolved token prices**. Both are open for discussion. The window is an expiry deadline, not a demonstrated completion guarantee under sustained online load.

### Proposed solution

The prototype changes the coordinator without changing the provider binary or WebSocket protocol:

- **API:** OpenAI-style file upload and asynchronous `/v1/batches` endpoints, polling, cancellation and result files. It also supports inline `requests[]` submissions and a synchronous `service_tier: "batch"` extension. Full API compatibility is not claimed.
- **Admission:** batch requests use loaded slots with no provider waiting queue, bounded by live pending load and the quality-concurrency allowance minus one. They do not trigger cold loads, join the online wait queue or launch speculative hedges. Online health/calibration signals are separated from batch outcomes.
- **Scheduling:** a 1 Hz dispatcher adjusts per-slot concurrency using AIMD and orders work by estimated deadline urgency. Admission limits reduce interference; they do not preempt running batch requests or guarantee unchanged online latency.
- **Storage:** prompts and results are encrypted before writing blob files. Postgres stores job metadata, including filenames, custom IDs and user metadata; those fields are not encrypted by the blob store. The storage key is derived from the coordinator mnemonic with a separate HKDF domain.
- **Trust boundary:** the coordinator and executing provider see plaintext during processing. Optional `result_public_key` encryption protects stored outputs for the consumer, but the coordinator sees the result before sealing it. This is not a coordinator-blind execution design.
- **Billing:** the prototype applies the lane multiplier after price resolution, removes the online per-request minimum for batch traffic, and records the lane for accounting. Pricing and refund policy need maintainer agreement.

OpenRouter is a potential integration target. The inline form and synchronous extension are prototype interfaces, not evidence of an agreed OpenRouter provider contract or completed integration. I would like to validate that contract before claiming compatibility.

### Implementation and evidence

Review snapshot, **2026-09-11**: fork `2238cf273`; upstream `09953c2fb`. The fork has not incorporated 23 newer upstream commits. I found no asynchronous batch API in the public upstream code checked at that snapshot; please point me to overlapping roadmap work.

- [Architecture and limitations](https://github.com/rajagurunath/d-inference/blob/2238cf273/docs/architecture/batch-lane.md)
- [Consumer API guide](https://github.com/rajagurunath/d-inference/blob/2238cf273/docs/consumer/batch-api.md)
- [Initial benchmark](https://github.com/rajagurunath/d-inference/blob/2238cf273/docs/reports/2026-09-04-batch-coserve-benchmark.md) and [rerun](https://github.com/rajagurunath/d-inference/blob/2238cf273/docs/reports/2026-09-05-batch-coserve-benchmark-rerun.md)

Earlier local acceptance runs exercised real inference, both submission forms, cancellation, tenant isolation and restart recovery. The latest review reran focused batch/API race tests and the sealed-blob tests successfully; it did not repeat live end-to-end testing or the full Postgres suite.

Benchmarks used **one M4 Pro, one provider, Qwen3.5-0.8B, short requests and sequential phases**. Co-serving batch throughput was 1.00 items/s against a 1.153 items/s batch-only baseline in the first run. The rerun exceeded its measured baseline; that variability is not evidence of exceeding a capacity ceiling. Latency samples were approximately 70 served requests per phase, with some 429s. These are preliminary observations, not fleet-scale evidence, a stable p99 estimate or proof of meeting 24-hour deadlines.

### Remaining work before production use

The prototype is available for design review, with these known gaps:

- **Settlement and refunds:** result-storage failures can discard charged work; a failed database finish can cause another inference execution. Refund failures are logged without durable retry. Result settlement and refund reconciliation need durable, idempotent handling.
- **Dispatcher ownership:** startup requeues all in-flight items. Multiple active coordinators need leases/fencing, or explicit single-dispatcher enforcement, before sharing a batch store.
- **Model authorization:** pinned builds and alias-based key permissions need consistent behavior when aliases change between submission and dispatch.
- **Integration seams:** internal dispatch currently uses an `httptest` recorder around the chat handler. A shared execution entry point would be preferable. File-form batch claims also need per-model budgeting, and result retention needs reliable recovery across restarts.
- **Validation:** integrate current upstream changes, exercise storage/network failures and provider churn, and run longer multi-provider tests with realistic request lengths and load patterns.

### Background references

My earlier Tidal posts provide background on the motivation and scheduling approach: [LinkedIn post](https://www.linkedin.com/posts/rajagurunath_llminference-vllm-gpu-ugcPost-7497783008763080704-902f/) · [X post](https://x.com/rajaguru95/status/2092319762407895267).

These describe the broader Tidal project. The implementation evidence and limitations above apply specifically to this Darkbloom prototype; preserving online latency is a design objective, not a guarantee established by these preliminary tests.

### Feedback requested

Does opportunistic batch serving fit the roadmap? In particular, I would value guidance on the admission policy, pricing, retention/trust model and intended consumer API contract.

Alternatives include dedicated batch providers for stronger isolation, or provider-side scheduling for finer control over interference. The coordinator-only approach offers a smaller initial integration surface; its limitations above remain important.

If there is interest, I can submit focused PRs in dependency order: storage and recovery → API → lane admission → dispatcher → billing → documentation and integration tests. Each would include its own validation; this RFC is to agree scope before proposing the full change set.

Feedback and alternative approaches are welcome. @Gajesh2007 @0xkydo, does this fit Darkbloom’s roadmap, and what would you change before considering it for upstream?

Start with docs/architecture/batch-lane.md and docs/consumer/batch-api.md, then review the prototype fork and its focused batch/API race and sealed-blob tests. The RFC still requests roadmap and contract decisions; done would require an agreed scope, admission, storage, API, billing and recovery plan before implementation is split into focused PRs.

Written by the indexing model from the issue text.