Worker channel `ArgumentError` on `Phoenix.Tracker.pool_size/1` during shutdown
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
User story
As a Lightning operator, I want worker reconnect attempts during a pod restart to fail cleanly rather than raise, so that rolling restarts don't leave behind a trail of ArgumentError reports in error monitoring.
Details
When a Lightning node receives SIGTERM and begins shutting down, workers that try to (re)join the worker channel during the shutdown window crash with an ArgumentError from :ets.lookup/2. The named LightningWeb.WorkerPresence ETS table has already been torn down by the time the join arrives, but Cowboy is still accepting WebSocket frames for ~1 second after SIGTERM.
:ets.lookup/2 (ets)
Phoenix.Tracker.pool_size/1 (lib/phoenix/tracker.ex:361)
Phoenix.Tracker.track/5 (lib/phoenix/tracker.ex:158)
LightningWeb.WorkerChannel.join/3 (lib/lightning_web/channels/worker_channel.ex:24)
Phoenix.Channel.Server.channel_join/4 (lib/phoenix/channel/server.ex:391)
Phoenix.Channel.Server.handle_info/2 (lib/phoenix/channel/server.ex:305)
It only seems to surface noticeably on single-instance deployments. Multi-instance setups appear to mask it — likely because the load balancer diverts traffic to a healthy replica well before the shutting-down instance closes its listener, so the race window never sees a phx_join. On a single-instance deployment there's no other replica to route to, so the LB keeps sending frames at the shutting-down node and the race becomes observable.
Implementation notes
It's an OTP supervision-order gap on shutdown. Lightning.Supervisor is :one_for_one and currently declares LightningWeb.Endpoint ahead of both Lightning.Workflows.Presence and LightningWeb.WorkerPresence in lib/lightning/application.ex. Reverse-order termination therefore kills the Presence supervisors (and their named ETS tables) before the Endpoint's internal Plug.Cowboy.Drainer gets a chance to suspend the Ranch listener — so for ~1s after SIGTERM, Cowboy is still accepting phx_join frames against a dead table.
Fix direction: reorder the children list so both Presence supervisors are declared after LightningWeb.Endpoint (declared later → terminated earlier under reverse-order shutdown). The Endpoint's existing Plug.Cowboy.Drainer then suspends the listener first, closing the race without any new dependency. PubSub already precedes both Presence modules so the boot-order dependency is preserved. A small regression-guard test asserting the relative order in Lightning.Application's children list is worth shipping alongside.
Release notes
Internal correctness fix — eliminates a noisy ArgumentError that could fire briefly on every pod restart when the node is the sole instance behind a load balancer. No user-facing behaviour change.
User acceptance criteria
- A regression-guard test in
test/lightning/application_test.exs(or equivalent) assertsLightningWeb.Endpointis declared ahead of bothLightning.Workflows.PresenceandLightningWeb.WorkerPresenceinLightning.Application's children list. - Manual repro via IEx (
Supervisor.terminate_child/2on the application supervisor's Presence children followed by atrack_worker/3call) no longer raisesArgumentErrorafter the reorder.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/lightning/application.ex and inspect the Lightning.Supervisor children list, then read test/lightning/application_test.exs for the application-startup test patterns. Verify the Endpoint is declared ahead of both Presence supervisors, add the relative-order regression guard, and run the application tests plus the documented IEx reproduction to confirm shutdown no longer raises ArgumentError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100