OpenFn / OpenFn/apollo

Nothing bounds how long a run can take

Open
#636 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
5
Forks
10
Avg merge
2d 20h
Merged PRs (30d)
17

Description

Raising the socket idle timeout to 255s and adding a keepalive (#629, #630) means a stream no longer goes idle, so nothing on the Apollo side bounds how long a run can take. The only timers in platform/src now are the 5s SIGKILL escalation and the socket idle timer the keepalive deliberately defeats.

A Python service that wedges — an HTTP call with no timeout, a retry loop that never gives up — holds a socket, an interval and an interpreter for as long as the caller stays attached, and the keepalive is what keeps the caller attached. N concurrent requests to a wedged service pin N interpreters with no ceiling.

Cancellation covers the case where the caller leaves: the abort fires, the child gets SIGTERM. The gap is the caller that waits.

In practice Lightning bounds this from its side with APOLLO_REQUEST_TIMEOUT_MS, so today it is not unbounded — but that is Lightning defending itself, not Apollo. Any other client, or a change to that setting, and the ceiling is gone.

What it probably wants is a run deadline in bridge.ts: an AbortSignal.timeout(n) composed with the caller's signal, firing the same path cancellation already uses, with its own error type so it reads differently from a client disconnect.

The open question is the number. It has to sit above the slowest legitimate run — p99 of workflow generation is around 50-57s — and below whatever the callers use, or Apollo never gets to be the one that explains what happened.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in platform/src/bridge.ts and trace the existing caller-cancellation path, then review how the socket, interval, and interpreter are managed during a run. Define the run deadline and distinct timeout error with the maintainer, accounting for the stated p99 workflow duration. Done means a wedged run is bounded without changing client-disconnect behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.