ExportWorker holds Postgres connection across chunked export
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Context
A single Lightning.WorkOrders.ExportWorker job failed in the last 90 days
with a DBConnection.ConnectionError ("tcp recv: closed"):
(lightning) lib/lightning/workorders/export_worker.ex:175:
Lightning.WorkOrders.ExportWorker.process_logs_and_dataclips_chunk/2
The export args show a ~3-month window with no extra filters — likely a large
result set. The pool closed the connection while the worker was mid-stream.
Root smell
process_logs_and_dataclips_chunk/2 (line 175) processes the export via a
single Stream.chunk_while that holds a Postgres connection across each
chunk's disk-write work. Long workorder lists + slow IO = connection held past
the pool timeout.
Options to evaluate
- Smaller chunks
- Dedicated checkout (
Repo.checkout/2) - Paginated export that releases the connection between pages
- Hard cap on export size with a user-facing message
Priority
Low — 1 incident in 90 days. Filing for tracking; revisit on recurrence or as
part of broader history-export improvements.
Note
The original issue body also called out two Sentry events for this single
incident. That turned out to be a generic pattern affecting every non-AI-queue
Oban error (Logger.error + explicit capture_exception), not an
ExportWorker-specific bug — it has been moved to #4747.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading lib/lightning/workorders/export_worker.ex at process_logs_and_dataclips_chunk/2, then trace how Stream.chunk_while handles database access and disk writes. Evaluate the listed options and define a bounded export approach that does not hold a Postgres connection across slow chunk work, with a user-facing size limit if that option is chosen.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100