benbjohnson / benbjohnson/litestream

S3 manifest: non-manifest-aware writers can silently stale an enabled manifest

Open
#1,391 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.4k
Forks
417
Avg merge
7d 1h
Merged PRs (30d)
21

Description

Follow-up to the S3 manifest feature (PR #1173 / #1172). Does not block that PR.

## Problem

The manifest consistency protocol assumes every writer against a replica is manifest-aware: each mutation acquires the `.manifest/lock.json` lease, writes a version-0 invalidation sentinel, and republishes a fenced manifest.

A writer that does not have the manifest capability enabled skips all of that (`ReplicaClient.WriteLTXFile` / `DeleteLTXFiles` short-circuit when all manifest flags are false) and mutates LTX objects without invalidating an existing `manifest.json`. A manifest-reading replica would then trust a stale manifest that is missing the new file (or still lists a deleted one) until the next manifest-aware mutation.

Concrete path: the `litestream-vfs` write path builds its replica client via `NewReplicaClientFromURL` and passes it to `NewVFS` without enabling any manifest flag, then calls `WriteLTXFile`. If that client writes to a bucket whose manifest is enabled by another (manifest-aware) writer, it can stale the manifest.

## Options

- Document explicitly (done in the PR #1173 review): do not mix a non-manifest-aware writer with a manifest-enabled replica.
- Consider a guard: a manifest-aware client could detect concurrent non-participating writes (e.g. an unexpected object count vs. manifest) and fall back to LIST, or a writer could refuse when it sees a foreign manifest it is not maintaining.
- Longer term, making manifest support available to every client type would remove the asymmetry.

## Severity

Low in practice (mixing a VFS writer with a manifest-enabled S3 replica is an unusual config), but it is a silent-correctness footgun worth tracking.

Contributor guide

Open the contributing guide

Research direction

Start with ReplicaClient.WriteLTXFile, DeleteLTXFiles, NewReplicaClientFromURL, and NewVFS, then read the manifest protocol around .manifest/lock.json. The issue has no selected implementation: determine whether the outcome is a restriction, a guard or fallback, or broader client support. Done means the chosen approach prevents or clearly documents stale manifests when writers are mixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.