Layr-Labs / Layr-Labs/d-inference
Investigate base-reward decline on healthy providers; expose zero-allocation reasons
- Dominant language
- Go
- Stars
- 522
- Forks
- 113
- Avg merge
- 17h 26m
- Merged PRs (30d)
- 111
Description
### Observed
My base rewards dropped from a historical average of approximately **$4.37/day** to **$0.50–$0.61/24h**, while paid inference continues. The historical average is my observation, not yet reconciled against settlement records.
Fleet: **3× M3 Ultra 256 GB + 2× M4 Max 128 GB**, all running **v0.9.0**, serving `gemma-4-26b-qat-4bit`. My M5 is intentionally offline.
Account pseudonym: `beamy-puppy-4259`.
### Evidence — September 9, 2026 UTC
- Coordinator: correct hardware/memory, hardware trust, MDM verification and Gemma advertisements on all five providers.
- Local state: Gemma warm; completed-request counters increasing.
- Reconstructed provider memory pressure: **16.8–25.9%**, below the reward engine’s **80%** exclusion threshold.
- Seven overlapping `/v1/provider/account-earnings?limit=1000` reads covered approximately **10:09–10:17**: paid Gemma entries continued, but no positive `base_reward` entries appeared.
- **12:31:55** leaderboard snapshot: **$0.509736 rewards**, **$13.362707 work**, rolling 24h.
- Network-wide rolling rewards remained approximately **$299–$300** during inspected hours.
### Relevant implementation
[`baserewards/alloc.go`](https://github.com/Layr-Labs/d-inference/blob/master/coordinator/payments/baserewards/alloc.go) prioritizes 48–96 GB providers, ranks by earned-to-floor coverage, and breaks equal-score ties by fixed provider-key order. Under budget exhaustion, unchanged rankings can repeatedly produce zero grants for eligible providers.
This behavior dates to [#282](https://github.com/Layr-Labs/d-inference/pull/282). It is a possible explanation, **not a confirmed diagnosis for my fleet**.
### Requested diagnosis
Please inspect my providers’ closed epochs around **September 9, 10:05–10:20 UTC**:
1. Candidate inclusion or exact failed eligibility gate.
2. `provider_floor_draws`: `floor_micro_usd`, `earned_micro_usd`, `amount_micro_usd`, `uptime_frac`, `memory_gb`.
3. Effective pool budget, reduction factor `k`, reserve fraction and account cap.
4. For zero grants: budget exhaustion and allocation ordering.
5. For positive grants: reconciliation with `provider_earnings`, ledger credits and leaderboard totals.
I recognize that the capped pool does not guarantee the nominal floor. I need to distinguish intentional zero allocation from eligibility or settlement failure. Please expose that distinction to providers.
Contributor guide
Research direction
Start with baserewards/alloc.go, then inspect the providers’ closed epochs around September 9, 10:05–10:20 UTC and the provider account-earnings records. Compare eligibility, floor draws, pool budget and allocation ordering with provider earnings, ledger credits and leaderboard totals; done means distinguishing intentional zero allocation from eligibility or settlement failure and exposing that reason to providers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, payments
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100