electric-sql / electric-sql/electric
AsyncDeleter deadlocks on full disk: crashes on boot with misleading :enoent, never reclaims trash
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Summary
`Electric.AsyncDeleter` cannot start when its trash directory's filesystem is full (or otherwise un-writable), and because `AsyncDeleter` is the process responsible for *emptying* the trash, a full disk becomes a **self-reinforcing deadlock**: the disk is full → `AsyncDeleter` crashes on boot → trash is never reclaimed → the disk stays full. On a multi-tenant deployment this takes the whole stack down and the affected tenant cannot recover on its own.
To make matters worse, the surfaced error is misleading — it reports `:enoent` on a path that the same function tried to create two lines earlier, so the real cause (`ENOSPC`) is hidden.
## Root cause
In `init/1` (`packages/sync-service/lib/electric/async_deleter.ex`):
```elixir
trash_dir = trash_dir!(stack_id)
File.mkdir_p(trash_dir) # return value is ignored
state = %__MODULE__{
stack_id: stack_id,
interval_ms: Keyword.get(opts, :cleanup_interval_ms, @default_cleanup_interval_ms),
pending: File.ls!(trash_dir) # raises here
}
```
`File.mkdir_p/1`'s return value is discarded. When the filesystem is full, `mkdir_p` fails with `{:error, :enospc}` and the directory is never created, so the immediately following `File.ls!(trash_dir)` raises `File.Error` with `reason: :enoent` (because the dir doesn't exist) — masking the real `ENOSPC`.
Since `AsyncDeleter` is a child of the stack supervisor, this crash propagates as `:failed_to_start_child` and brings down the entire stack on boot.
## Observed failure
Seen on a multi-tenant deployment when `/var/electric` hit 100% usage:
```
{:shutdown,
{:failed_to_start_child, Electric.AsyncDeleter,
{%File.Error{reason: :enoent,
path: "/var/electric//shapes/.electric_trash/",
action: "list directory"},
[{File, :ls!, 1, [file: ~c"lib/file.ex", line: 1700]},
{Electric.AsyncDeleter, :init, 1, [file: ~c"lib/electric/async_deleter.ex", line: 84]},
...]}}}
```
`df -h` on the host at the time:
```
/dev/nvme1n1 20G 20G 20K 100% /var/electric
```
The stack went into a crash/cleanup/retry loop and the tenant could not initialize until the volume was freed manually.
## Why this is especially bad
`AsyncDeleter` is the GC mechanism for batched deletes. The one condition under which it's most needed — a full disk with reclaimable trash sitting in the trash dir — is exactly the condition under which it refuses to start, so it can never reclaim that trash. Recovery requires out-of-band manual deletion of `*/shapes/.electric_trash/*` (and any `*/trash/*`).
## Suggested fix
1. **Don't ignore `File.mkdir_p/1`'s result** — handle the error and fail with the real reason (or log it) instead of letting a later `File.ls!` raise a misleading `:enoent`.
2. **Don't hard-crash `init` on a transient/full-disk condition.** Two reasonable options:
- Treat a non-listable/absent trash dir as "nothing pending" (`pending: []`) and let the periodic cleanup retry, so the process can at least boot and start reclaiming once space frees up; or
- Make the boot resilient (e.g. `File.ls/1` with a fallback) so a full disk doesn't take down the whole stack.
Either way, the key goal is to break the deadlock so that `AsyncDeleter` can come up and do its job (freeing trash) precisely when the disk is full.
## Environment
- Observed against sync-service at commit `12a6ee0` (the relevant `init/1` code is unchanged across recent revisions).
- Multi-tenant embedded usage (each tenant is a stack).
Contributor guide
Assessment
This issue has not been assessed yet.