apache / apache/texera

Two retry loops still hand-rolled: URLFetchUtil retries without delay, PythonProxyClient waits a constant delay

Open
#7,124 1 comment 0 reactions 1 assignee Claimed by @anthonychengit View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Two retry loops were left hand-rolled when #7119 consolidated the rest onto `RetryUtil.withBackoff`, because converting them changes runtime behavior rather than just structure. Each needs a decision before it can move.

**1. `URLFetchUtil.getInputStreamFromURL`** (`common/workflow-operator`)

```scala
def getInputStreamFromURL(urlObj: URL, retries: Int = 5): Option[InputStream]
```

- Retries 5x with **no delay at all**, and returns `Option` rather than throwing.
- Adopting the util's backoff means a dead URL costs ~3 s (200+400+800+1600) before giving up, where today it fails immediately. That is arguably the fix — an unthrottled 5x retry against a remote host is a small retry storm — but it changes how long a URL-scan operator takes to fail.
- The `Option` return is a one-line adapter (`Try(...).toOption`), so the delay is the only real question.

**2. `PythonProxyClient`'s Flight-connect loop** (`amber`)

```scala
while (!connected && tryCount <= MAX_TRY_COUNT) { ... Thread.sleep(UNIT_WAIT_TIME_MS) ... }
```

- Waits a **constant** `UNIT_WAIT_TIME_MS` between attempts, not a doubling one, so the util needs a delay-multiplier knob (`1` = constant) before it can host this.
- Closes the Flight client between attempts; that per-attempt cleanup has to move into the operation body.
- Throws `WorkflowRuntimeException` on give-up, where the util wraps in `RuntimeException`. Either the util grows a give-up hook, or this call site accepts the different type — worth deciding deliberately, since worker-bring-up failures are surfaced to users.

Both are candidates, not obligations: if the answer for either is "the current behavior is what we want", closing this with that note recorded is a fine outcome.

### Task Type

- [x] Refactor / Cleanup

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.