apple / apple/foundationdb

AsyncFileKAIO::truncate treats interrupted fallocate as fatal io_error

Open
#14,041 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

## Observed failure

FoundationDB 7.3.77 (`3ea44ce1d9003ad095e408039e1f755c319c4dfb`, official `foundationdb/foundationdb:7.3.77` image) exited during a normal single-node `memory`-engine integration run on Linux. Docker recorded exit code 1 and `OOMKilled=false`. The data directory was on tmpfs. The server had been running for 72 seconds; it was not intentionally stopped.

The preserved trace shows this sequence (irrelevant identifiers omitted):

```text
AsyncFileKAIOAllocateError
Filename=/var/fdb/data/logqueue-V_6-...-1.fdq
Size=101208064 UnixErrorCode=4 UnixError="Interrupted system call"
AsyncFileKAIOAllocateError
Filename=/var/fdb/data/logqueue-V_6-...-1.fdq
Size=90726400 UnixErrorCode=4 UnixError="Interrupted system call"
RDQPushAndCommitError Error=io_error ErrorCode=1510
SharedTLogFailed Error=io_error
WorkerFailed Error=io_error
StopAfterError Error=io_error
```

Stdout: `Fatal Error: Disk i/o operation failed`.

The trace establishes EINTR and its fatal propagation, not which signal interrupted the syscall.

## Source

In tag 7.3.77, `fdbrpc/include/fdbrpc/AsyncFileKAIO.actor.h`, `AsyncFileKAIO::truncate` calls `fallocate(fd, 0, 0, size)` once. If it fails, only `EOPNOTSUPP` gets fallback handling; every other errno, including `EINTR`, becomes `io_error()`. The current main version at `fdbrpc/AsyncFileKAIO.h` has the same branch.

Could this path retry interrupted calls, retaining the existing handling for genuine I/O errors? `ftruncate` immediately below also has no EINTR retry, although the observed failure above was specifically `fallocate`.

## Deterministic verification / downstream workaround

We added a real-container regression that installs a **test-only, container-local** seccomp rule returning errno 4 on `fallocate` (no Go-client mock). With the default KAIO backend, both tmpfs and on-disk fixtures fail initialization because the server exits. With `--knob_disable_posix_kernel_aio=1`, both initialize and successfully commit/read a value. An actual `fallocate` command inside each successful container still returns `Interrupted system call`, proving the injected fault remains active. Explicitly restoring the knob to 0 reproduces the fatal disk error again.

The downstream fixture workaround uses the supported EIO backend, whose file growth dispatches `eio_ftruncate` instead of this `fallocate` path. It does not recreate an in-use database, ignore I/O failures, or disable durability. The normal-run failure and permanent regression are being recorded in birdayz/fdb-go#780 (RFC-250).

Contributor guide

Open the contributing guide

Research direction

Read fdbrpc/AsyncFileKAIO.h at AsyncFileKAIO::truncate, then inspect the adjacent ftruncate path and existing EOPNOTSUPP handling. Use the deterministic container regression described in the issue to verify that an interrupted fallocate no longer causes fatal initialization, while genuine I/O errors still propagate as io_error().

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.