electric-sql / electric-sql/electric

Automated fuzz-testing harness for transaction processing pipeline

Open
#3,821 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Background

After the direct-to-storage streaming of transaction fragments implemented in https://github.com/electric-sql/electric/pull/3783, it has become increasingly difficult to be fully confident that all edge cases are covered on the path from Postgres to the shape log. An automated test harness that can generate test data and inject faults into consumer processes would help us gain that confidence by exercising various scenarios where transaction processing gets interrupted at different points in the pipeline.

## Proposed design

- Electric starts in single-stack mode connected to an empty database.
- Writes to storage are deliberately slowed down to leave room for a sufficient number of concurrent clients to fuzz-load the read API. This can be achieved in one of two ways:
- A wrapper around storage calls that introduces artificial delays before invoking the underlying storage functions.
- Configuring Electric to use a filesystem interface that throttles file operations.
- Many concurrent clients create shapes, requesting not only valid offsets obtained from the server but also probing "in-between" offsets: random values between any two known transaction boundaries, as well as offsets beyond the current log head ("in the future").

## Invariants to validate

With all pieces in place, a random write load is continuously generated against the database while shape requests arrive from clients. The test harness should run the following checks throughout the duration of the test.

### 1. Transaction boundaries are maintained

Regardless of which offset a client specifies in its request, the server must always return one of the following:

- A valid set of changes covering one or more **complete** transactions.
- An error indicating that the supplied offset is invalid.
- An `up-to-date` control message.

At no point should any client observe incomplete changes from a transaction. The invariant that the last operation in any transaction includes the `last=true` header must hold at all times.

### 2. No missing data

From any single client's perspective, the observed sequence of operations must match the order in which they were committed to the database, excluding operations that fall outside the client's shape definition.

At the end of a test run — once write load on the database has stopped — every client must have obtained all data that was inserted into the database and matches its shape.

### 3. No duplicate writes

In addition to storage slowdowns, the Electric server should randomly crash shape consumer processes and let them restart and resume transaction processing. Beyond the checks above, this scenario validates that reprocessing a transaction that was already partially processed and/or written to the shape log does not result in duplicate transaction fragments in storage.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.