electric-sql / electric-sql/electric

AsyncDeleter deadlocks on full disk: crashes on boot with misleading :enoent, never reclaims trash

Open
#4,595 0 comments 0 reactions 1 assignee Claimed by @alco View on GitHub
reliability
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Summary

`Electric.AsyncDeleter` cannot start when its trash directory's filesystem is full (or otherwise un-writable), and because `AsyncDeleter` is the process responsible for *emptying* the trash, a full disk becomes a **self-reinforcing deadlock**: the disk is full → `AsyncDeleter` crashes on boot → trash is never reclaimed → the disk stays full. On a multi-tenant deployment this takes the whole stack down and the affected tenant cannot recover on its own.

To make matters worse, the surfaced error is misleading — it reports `:enoent` on a path that the same function tried to create two lines earlier, so the real cause (`ENOSPC`) is hidden.

## Root cause

In `init/1` (`packages/sync-service/lib/electric/async_deleter.ex`):

```elixir
trash_dir = trash_dir!(stack_id)
File.mkdir_p(trash_dir) # return value is ignored

state = %__MODULE__{
stack_id: stack_id,
interval_ms: Keyword.get(opts, :cleanup_interval_ms, @default_cleanup_interval_ms),
pending: File.ls!(trash_dir) # raises here
}
```

`File.mkdir_p/1`'s return value is discarded. When the filesystem is full, `mkdir_p` fails with `{:error, :enospc}` and the directory is never created, so the immediately following `File.ls!(trash_dir)` raises `File.Error` with `reason: :enoent` (because the dir doesn't exist) — masking the real `ENOSPC`.

Since `AsyncDeleter` is a child of the stack supervisor, this crash propagates as `:failed_to_start_child` and brings down the entire stack on boot.

## Observed failure

Seen on a multi-tenant deployment when `/var/electric` hit 100% usage:

```
{:shutdown,
{:failed_to_start_child, Electric.AsyncDeleter,
{%File.Error{reason: :enoent,
path: "/var/electric//shapes/.electric_trash/",
action: "list directory"},
[{File, :ls!, 1, [file: ~c"lib/file.ex", line: 1700]},
{Electric.AsyncDeleter, :init, 1, [file: ~c"lib/electric/async_deleter.ex", line: 84]},
...]}}}
```

`df -h` on the host at the time:

```
/dev/nvme1n1 20G 20G 20K 100% /var/electric
```

The stack went into a crash/cleanup/retry loop and the tenant could not initialize until the volume was freed manually.

## Why this is especially bad

`AsyncDeleter` is the GC mechanism for batched deletes. The one condition under which it's most needed — a full disk with reclaimable trash sitting in the trash dir — is exactly the condition under which it refuses to start, so it can never reclaim that trash. Recovery requires out-of-band manual deletion of `*/shapes/.electric_trash/*` (and any `*/trash/*`).

## Suggested fix

1. **Don't ignore `File.mkdir_p/1`'s result** — handle the error and fail with the real reason (or log it) instead of letting a later `File.ls!` raise a misleading `:enoent`.
2. **Don't hard-crash `init` on a transient/full-disk condition.** Two reasonable options:
- Treat a non-listable/absent trash dir as "nothing pending" (`pending: []`) and let the periodic cleanup retry, so the process can at least boot and start reclaiming once space frees up; or
- Make the boot resilient (e.g. `File.ls/1` with a fallback) so a full disk doesn't take down the whole stack.

Either way, the key goal is to break the deadlock so that `AsyncDeleter` can come up and do its job (freeing trash) precisely when the disk is full.

## Environment

- Observed against sync-service at commit `12a6ee0` (the relevant `init/1` code is unchanged across recent revisions).
- Multi-tenant embedded usage (each tenant is a stack).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.