OpenFn / OpenFn/lightning

ExportWorker holds Postgres connection across chunked export

Open
#4,749 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

architecture needs investigation oban errors
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Context

A single Lightning.WorkOrders.ExportWorker job failed in the last 90 days
with a DBConnection.ConnectionError ("tcp recv: closed"):

(lightning) lib/lightning/workorders/export_worker.ex:175:
    Lightning.WorkOrders.ExportWorker.process_logs_and_dataclips_chunk/2

The export args show a ~3-month window with no extra filters — likely a large
result set. The pool closed the connection while the worker was mid-stream.

Root smell

process_logs_and_dataclips_chunk/2 (line 175) processes the export via a
single Stream.chunk_while that holds a Postgres connection across each
chunk's disk-write work. Long workorder lists + slow IO = connection held past
the pool timeout.

Options to evaluate

  • Smaller chunks
  • Dedicated checkout (Repo.checkout/2)
  • Paginated export that releases the connection between pages
  • Hard cap on export size with a user-facing message

Priority

Low — 1 incident in 90 days. Filing for tracking; revisit on recurrence or as
part of broader history-export improvements.

Note

The original issue body also called out two Sentry events for this single
incident. That turned out to be a generic pattern affecting every non-AI-queue
Oban error (Logger.error + explicit capture_exception), not an
ExportWorker-specific bug — it has been moved to #4747.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading lib/lightning/workorders/export_worker.ex at process_logs_and_dataclips_chunk/2, then trace how Stream.chunk_while handles database access and disk writes. Evaluate the listed options and define a bounded export approach that does not hold a Postgres connection across slow chunk work, with a user-facing size limit if that option is chosen.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.