Layr-Labs / Layr-Labs/d-inference

Investigate base-reward decline on healthy providers; expose zero-allocation reasons

Open
#878 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
522
Forks
113
Avg merge
17h 26m
Merged PRs (30d)
111

Description

### Observed

My base rewards dropped from a historical average of approximately **$4.37/day** to **$0.50–$0.61/24h**, while paid inference continues. The historical average is my observation, not yet reconciled against settlement records.

Fleet: **3× M3 Ultra 256 GB + 2× M4 Max 128 GB**, all running **v0.9.0**, serving `gemma-4-26b-qat-4bit`. My M5 is intentionally offline.

Account pseudonym: `beamy-puppy-4259`.

### Evidence — September 9, 2026 UTC

- Coordinator: correct hardware/memory, hardware trust, MDM verification and Gemma advertisements on all five providers.
- Local state: Gemma warm; completed-request counters increasing.
- Reconstructed provider memory pressure: **16.8–25.9%**, below the reward engine’s **80%** exclusion threshold.
- Seven overlapping `/v1/provider/account-earnings?limit=1000` reads covered approximately **10:09–10:17**: paid Gemma entries continued, but no positive `base_reward` entries appeared.
- **12:31:55** leaderboard snapshot: **$0.509736 rewards**, **$13.362707 work**, rolling 24h.
- Network-wide rolling rewards remained approximately **$299–$300** during inspected hours.

### Relevant implementation

[`baserewards/alloc.go`](https://github.com/Layr-Labs/d-inference/blob/master/coordinator/payments/baserewards/alloc.go) prioritizes 48–96 GB providers, ranks by earned-to-floor coverage, and breaks equal-score ties by fixed provider-key order. Under budget exhaustion, unchanged rankings can repeatedly produce zero grants for eligible providers.

This behavior dates to [#282](https://github.com/Layr-Labs/d-inference/pull/282). It is a possible explanation, **not a confirmed diagnosis for my fleet**.

### Requested diagnosis

Please inspect my providers’ closed epochs around **September 9, 10:05–10:20 UTC**:

1. Candidate inclusion or exact failed eligibility gate.
2. `provider_floor_draws`: `floor_micro_usd`, `earned_micro_usd`, `amount_micro_usd`, `uptime_frac`, `memory_gb`.
3. Effective pool budget, reduction factor `k`, reserve fraction and account cap.
4. For zero grants: budget exhaustion and allocation ordering.
5. For positive grants: reconciliation with `provider_earnings`, ledger credits and leaderboard totals.

I recognize that the capped pool does not guarantee the nominal floor. I need to distinguish intentional zero allocation from eligibility or settlement failure. Please expose that distinction to providers.

Contributor guide

Open the contributing guide

Research direction

Start with baserewards/alloc.go, then inspect the providers’ closed epochs around September 9, 10:05–10:20 UTC and the provider account-earnings records. Compare eligibility, floor draws, pool budget and allocation ordering with provider earnings, ledger credits and leaderboard totals; done means distinguishing intentional zero allocation from eligibility or settlement failure and exposing that reason to providers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, payments
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.