apache / apache/texera

HuggingFace inference operator: remember the working provider instead of retrying dead ones every row

Open
#7,202 3 comments 0 reactions 1 assignee Claimed by @juliethecao View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
2d 8h
Merged PRs (30d)
209

Description

### Task Summary

`_post_with_fallback` walks the full provider list for every row, so a dead first provider costs a failed request (up to a 120s timeout) per row.

### Context
There is no memory of which provider last succeeded. On a 1,000-row table a dead cheapest-provider means ~1,000 wasted attempts; Replicate/Wavespeed polling compounds it (up to ~10 min/row).

### Proposed change
Cache the first provider that returns success and try it first (or exclusively) on subsequent rows, falling back to the full walk only if it starts failing.

### Required test
Behavioral: a second row does not re-probe a provider already known to be dead / prefers the last-successful provider.

### Related
Precedes the provider cost-ranking enhancement.

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [x] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.