OpenFn / OpenFn/lightning

Worker channel `ArgumentError` on `Phoenix.Tracker.pool_size/1` during shutdown

Open
#4,805 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Monitoring Sentry
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

User story

As a Lightning operator, I want worker reconnect attempts during a pod restart to fail cleanly rather than raise, so that rolling restarts don't leave behind a trail of ArgumentError reports in error monitoring.

Details

When a Lightning node receives SIGTERM and begins shutting down, workers that try to (re)join the worker channel during the shutdown window crash with an ArgumentError from :ets.lookup/2. The named LightningWeb.WorkerPresence ETS table has already been torn down by the time the join arrives, but Cowboy is still accepting WebSocket frames for ~1 second after SIGTERM.

:ets.lookup/2 (ets)
Phoenix.Tracker.pool_size/1            (lib/phoenix/tracker.ex:361)
Phoenix.Tracker.track/5                (lib/phoenix/tracker.ex:158)
LightningWeb.WorkerChannel.join/3      (lib/lightning_web/channels/worker_channel.ex:24)
Phoenix.Channel.Server.channel_join/4  (lib/phoenix/channel/server.ex:391)
Phoenix.Channel.Server.handle_info/2   (lib/phoenix/channel/server.ex:305)

It only seems to surface noticeably on single-instance deployments. Multi-instance setups appear to mask it — likely because the load balancer diverts traffic to a healthy replica well before the shutting-down instance closes its listener, so the race window never sees a phx_join. On a single-instance deployment there's no other replica to route to, so the LB keeps sending frames at the shutting-down node and the race becomes observable.

Implementation notes

It's an OTP supervision-order gap on shutdown. Lightning.Supervisor is :one_for_one and currently declares LightningWeb.Endpoint ahead of both Lightning.Workflows.Presence and LightningWeb.WorkerPresence in lib/lightning/application.ex. Reverse-order termination therefore kills the Presence supervisors (and their named ETS tables) before the Endpoint's internal Plug.Cowboy.Drainer gets a chance to suspend the Ranch listener — so for ~1s after SIGTERM, Cowboy is still accepting phx_join frames against a dead table.

Fix direction: reorder the children list so both Presence supervisors are declared after LightningWeb.Endpoint (declared later → terminated earlier under reverse-order shutdown). The Endpoint's existing Plug.Cowboy.Drainer then suspends the listener first, closing the race without any new dependency. PubSub already precedes both Presence modules so the boot-order dependency is preserved. A small regression-guard test asserting the relative order in Lightning.Application's children list is worth shipping alongside.

Release notes

Internal correctness fix — eliminates a noisy ArgumentError that could fire briefly on every pod restart when the node is the sole instance behind a load balancer. No user-facing behaviour change.

User acceptance criteria
  • A regression-guard test in test/lightning/application_test.exs (or equivalent) asserts LightningWeb.Endpoint is declared ahead of both Lightning.Workflows.Presence and LightningWeb.WorkerPresence in Lightning.Application's children list.
  • Manual repro via IEx (Supervisor.terminate_child/2 on the application supervisor's Presence children followed by a track_worker/3 call) no longer raises ArgumentError after the reorder.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/lightning/application.ex and inspect the Lightning.Supervisor children list, then read test/lightning/application_test.exs for the application-startup test patterns. Verify the Endpoint is declared ahead of both Presence supervisors, add the relative-order regression guard, and run the application tests plus the documented IEx reproduction to confirm shutdown no longer raises ArgumentError.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.