HuggingFace inference operator: remember the working provider instead of retrying dead ones every row
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 209
Description
### Task Summary
`_post_with_fallback` walks the full provider list for every row, so a dead first provider costs a failed request (up to a 120s timeout) per row.
### Context
There is no memory of which provider last succeeded. On a 1,000-row table a dead cheapest-provider means ~1,000 wasted attempts; Replicate/Wavespeed polling compounds it (up to ~10 min/row).
### Proposed change
Cache the first provider that returns success and try it first (or exclusively) on subsequent rows, falling back to the full walk only if it starts failing.
### Required test
Behavioral: a second row does not re-probe a provider already known to be dead / prefers the last-successful provider.
### Related
Precedes the provider cost-ranking enhancement.
### Task Type
- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [x] Performance
- [ ] Other
Contributor guide
Assessment
This issue has not been assessed yet.