feat(connectors): shared retry_async helper in iggy_connector_sdk::retry
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
The connectors codebase has three hand-written copies of the same retry skeleton: `HttpRetryMiddleware::handle` and `check_connectivity_with_retry` in `core/connectors/sdk/src/retry.rs`, and `DorisSink::load_batch` in `core/connectors/sinks/doris_sink/src/lib.rs`.
The Doris sink cannot reuse the middleware: Doris Stream Load signals failure as HTTP 200 with `{"Status":"Fail"}` in the body, and the middleware never inspects response bodies. Any connector whose backend reports errors in-band has the same problem, so each one ends up hand-rolling the loop.
The copies have already diverged on backoff convention: the SDK's two loops pass the already-incremented attempt to `exponential_backoff`, so their first retry waits `2 * base`, while the Doris sink passes `attempt - 1` so the first retry waits the configured base delay, matching the documented meaning of `retry_delay`.
Raised as a non-blocking follow-up in the review of #3574: https://github.com/apache/iggy/pull/3574#discussion_r3507968371
### Affected area / component
Connectors
### Proposed solution
Expose a generic `retry_async(operation, policy)` in `iggy_connector_sdk::retry` that owns attempt counting, the transient-error predicate, capped exponential backoff with jitter, and per-retry logging, then collapse the three call sites onto it.
The shared helper should keep the base-delay-first convention (`base * 2^(k-1)` for retry `k`), since that is what the `retry_delay` config field documents.
### Alternatives considered
- Reuse `HttpRetryMiddleware` in the Doris sink: not possible, the middleware classifies on status codes and never inspects response bodies, so an in-band `{"Status":"Fail"}` looks like success to it.
- Keep per-connector loops: the exponent divergence above shows the conventions drift as soon as the skeleton is copied.
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.