OpenFn / OpenFn/kit

Worker: after DB timeouts in lightning, the connection doesn't seem to restore

Open
#959 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DevX
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

From speaking with MSF it seems like when sockets start timing out because of load on the app's postgres DB, the socket drops its connection and then doesn't manage to reconnect.

My expectation is that after a socket timeout, maybe the run will be lost (maybe!), but the claim loop should happily reconnect and start claiming as soon as the DB is active.

Not sure how to repro this - maybe a very low db timeout in the app.

Options:

  • The reconnect doesn't work in the phoenix websocket (unlikely, I'm sure I've seen this work and may even have tests)
  • The JWT has expired or something which is preventing the reconnect (I'd expect an error rather than a timeout)
  • Maybe the DB is just continually busy and the app is genuinely unable to respond in time

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the worker claim loop and the Phoenix websocket reconnect path described in the issue, then examine how PostgreSQL socket timeouts are surfaced. Try reproducing the failure with a very low database timeout while the app is under load. Done means the claim loop reconnects and resumes claiming after the database becomes responsive, with the failure mode distinguished from JWT expiry or sustained database unavailability.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.