AsyncFileKAIO::truncate treats interrupted fallocate as fatal io_error
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
## Observed failure
FoundationDB 7.3.77 (`3ea44ce1d9003ad095e408039e1f755c319c4dfb`, official `foundationdb/foundationdb:7.3.77` image) exited during a normal single-node `memory`-engine integration run on Linux. Docker recorded exit code 1 and `OOMKilled=false`. The data directory was on tmpfs. The server had been running for 72 seconds; it was not intentionally stopped.
The preserved trace shows this sequence (irrelevant identifiers omitted):
```text
AsyncFileKAIOAllocateError
Filename=/var/fdb/data/logqueue-V_6-...-1.fdq
Size=101208064 UnixErrorCode=4 UnixError="Interrupted system call"
AsyncFileKAIOAllocateError
Filename=/var/fdb/data/logqueue-V_6-...-1.fdq
Size=90726400 UnixErrorCode=4 UnixError="Interrupted system call"
RDQPushAndCommitError Error=io_error ErrorCode=1510
SharedTLogFailed Error=io_error
WorkerFailed Error=io_error
StopAfterError Error=io_error
```
Stdout: `Fatal Error: Disk i/o operation failed`.
The trace establishes EINTR and its fatal propagation, not which signal interrupted the syscall.
## Source
In tag 7.3.77, `fdbrpc/include/fdbrpc/AsyncFileKAIO.actor.h`, `AsyncFileKAIO::truncate` calls `fallocate(fd, 0, 0, size)` once. If it fails, only `EOPNOTSUPP` gets fallback handling; every other errno, including `EINTR`, becomes `io_error()`. The current main version at `fdbrpc/AsyncFileKAIO.h` has the same branch.
Could this path retry interrupted calls, retaining the existing handling for genuine I/O errors? `ftruncate` immediately below also has no EINTR retry, although the observed failure above was specifically `fallocate`.
## Deterministic verification / downstream workaround
We added a real-container regression that installs a **test-only, container-local** seccomp rule returning errno 4 on `fallocate` (no Go-client mock). With the default KAIO backend, both tmpfs and on-disk fixtures fail initialization because the server exits. With `--knob_disable_posix_kernel_aio=1`, both initialize and successfully commit/read a value. An actual `fallocate` command inside each successful container still returns `Interrupted system call`, proving the injected fault remains active. Explicitly restoring the knob to 0 reproduces the fatal disk error again.
The downstream fixture workaround uses the supported EIO backend, whose file growth dispatches `eio_ftruncate` instead of this `fallocate` path. It does not recreate an in-use database, ignore I/O failures, or disable durability. The normal-run failure and permanent regression are being recorded in birdayz/fdb-go#780 (RFC-250).
Contributor guide
Research direction
Read fdbrpc/AsyncFileKAIO.h at AsyncFileKAIO::truncate, then inspect the adjacent ftruncate path and existing EOPNOTSUPP handling. Use the deterministic container regression described in the issue to verify that an interrupted fallocate no longer causes fatal initialization, while genuine I/O errors still propagate as io_error().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100