OpenFn / OpenFn/lightning

UsageTracking failure - Postgrex.Protocol (#PID<0.4680.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.53147.0> (Lightning.UsageTracking.ReportWorker) timed out because it queued and checked out the connection for longer than 70000ms

Open
#4,282 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Sentry Issue: LIGHTNING-16F

Postgrex.Protocol (#PID<0.4680.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.53147.0> (Lightning.UsageTracking.ReportWorker) timed out because it queued and checked out the connection for longer than 70000ms

#PID<0.53147.0> (Lightning.UsageTracking.ReportWorker) was at location:

    (lightning 2.15.6) lib/lightning/usage_tracking/workflow_metrics_service.ex:69: anonymous fn/3 in Lightning.UsageTracking.WorkflowMetricsService.metric_updater/1
    (elixir 1.18.3) lib/enum.ex:2546: Enum."-reduce/3-lists^foldl/2-0-"/3
    (lightning 2.15.6) lib/lightning/repo.ex:2: anonymous fn/1 in Lightning.Repo.transaction/2
    (ecto 3.13.2) lib/ecto/repo/transaction.ex:7: anonymous fn/2 in Ecto.Repo.Transaction.transact/4
    (ecto_sql 3.13.2) lib/ecto/adapters/sql.ex:1458: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
    (db_connection 2.8.1) lib/db_connection.ex:1753: DBConnection.run_transaction/4
    (lightning 2.15.6) lib/lightning/usage_tracking/workflow_metrics_service.ex:29: Lightning.UsageTracking.WorkflowMetricsService.metrics_for_runs/2
    (lightning 2.15.6) lib/lightning/usage_tracking/workflow_metrics_service.ex:19: Lightning.UsageTracking.WorkflowMetricsService.generate_metrics/3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/lightning/usage_tracking/workflow_metrics_service.ex at lines 19, 29, and 69, following generate_metrics/3 through metrics_for_runs/2 and metric_updater/1. Use the Sentry report and Postgrex timeout details to identify the long connection checkout or transaction path; done means the UsageTracking.ReportWorker no longer disconnects after exceeding 70000ms.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.