electric-sql / electric-sql/electric
StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Background
`Electric.StatusMonitor.wait_until/3` is called from `Api.validate_params` → `hold_until_stack_ready` on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to `GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity)` and parks in the StatusMonitor's `waiters` MapSet until readiness is signalled.
This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with `:infinity` per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.
When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers `maybe_reply_to_waiters/1` to iterate every entry and `GenServer.reply/2` to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.
## Goal
Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.
## Design
### 1. A small reusable polling primitive
```elixir
defmodule Electric.PollWait do
@moduledoc """
Per-process bounded polling of a cheap (ETS-backed) condition.
Sleeps between checks with exponential backoff (doubling, capped) and
bounded jitter so concurrent waiters land on distinct ETS reads
instead of stampeding the same millisecond windows.
"""
@default_initial_interval 25
@default_max_interval 500
@default_backoff 2.0
@default_jitter 0.25
@type check :: (-> {:ready, term()} | :ready | :not_ready)
@spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
def until(check_fun, timeout, opts \\ []) do
initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
max = Keyword.get(opts, :max_interval, @default_max_interval)
factor = Keyword.get(opts, :backoff, @default_backoff)
jitter = Keyword.get(opts, :jitter, @default_jitter)
do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
end
defp deadline(:infinity), do: :infinity
defp deadline(t) when is_integer(t) and t >= 0,
do: System.monotonic_time(:millisecond) + t
defp do_until(check_fun, deadline, interval, max, factor, jitter) do
case check_fun.() do
:not_ready ->
case remaining(deadline) do
0 -> :timeout
rem ->
Process.sleep(min(jittered(interval, jitter), rem))
do_until(check_fun, deadline, min(round(interval * factor), max),
max, factor, jitter)
end
ready -> ready
end
end
defp jittered(interval, jitter) do
spread = max(1, round(interval * jitter))
interval + :rand.uniform(2 * spread + 1) - spread - 1
end
defp remaining(:infinity), do: :infinity
defp remaining(deadline),
do: max(0, deadline - System.monotonic_time(:millisecond))
end
```
Default backoff schedule: **25 → 50 → 100 → 200 → 400 → 500 → 500 …** ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.
### 2. Adaptive switch-over driven by a congestion flag
We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.
**Caller side:**
```elixir
def wait_until(stack_id, level, opts) do
case status_check(stack_id, level, opts) do
{:ready, value} ->
value
:not_ready ->
if congested?(stack_id),
do: poll_wait(stack_id, level, opts),
else: call_wait(stack_id, level, opts)
end
end
```
`congested?/1` is one ETS read on the same `StatusMonitor:` table that already backs the fast-path readiness check (already configured with `read_concurrency: true`).
**Server side:**
```elixir
@congested_threshold 100 # tunable
# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))
# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))
defp maybe_set_congested(state, size) when size >= @congested_threshold do
:ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok
defp maybe_clear_congested(state, 0) do
:ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok
```
## Costs
Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The `StatusMonitor` ETS table is already `read_concurrency: true` so concurrent reads scale.
Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially `GenServer.reply` to thousands of waiters.
## Reusability
Same `PollWait` + congestion-flag pattern slots into the EtsInspector follow-up (#4370):
- Polling: `PollWait.until/3` reuses verbatim.
- Trigger: `GenServer.cast(EtsInspector, {:ensure_filled, key})` to start the DB fill; cast is idempotent per key under server-side coalescing.
- Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.
So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.
## Suggested execution
1. Land `Electric.PollWait` with unit tests covering: ready/timeout/backoff schedule/jitter bounds.
2. Add the `@congested_key` flag + threshold logic to `StatusMonitor`.
3. Switch `wait_until/3` callers to the adaptive branch. Delete the now-unused `wait_until_async/2` if no callers remain.
4. Smoke-test the cold-start path: simulate N concurrent `wait_until` calls during a delayed `mark_*_ready` flip and assert the GenServer's reply burst is bounded by `@congested_threshold`.
Contributor guide
Assessment
This issue has not been assessed yet.