benbjohnson / benbjohnson/litestream

feat: Expose distributed leasing in replicate -exec lifecycle

Open
#1,301 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
14.4k
Forks
417
Avg merge
7d 1h
Merged PRs (30d)
21

Description

## Feature Description

Expose Litestream's distributed leasing support in `litestream.yml` and enforce it
inside `litestream replicate` / `replicate -exec`, so users can opt into a
fail-safe guard against multiple Litestream processes writing to the same replica
location.

Litestream already documents the caveat this would address:

> Multiple applications replicating into the same bucket & path can cause
> situations where you will be unable to restore. It is your responsibility to
> ensure you do not have multiple applications replicating concurrently.
>
> https://litestream.io/tips/#multiple-applications-replicating-into-location-can-corrupt

I searched related issues/PRs and found:

- #1073 — merged S3 distributed leasing infrastructure.
- #922 — earlier leasing design discussion, closed in favor of #1073.
- #1119 — Docker zero-downtime deploy question; the maintainer response says
leasing infrastructure exists but is not yet integrated into the replication
loop.
- #224 — restore failure caused by another machine writing to the same bucket
location; the discussion explicitly mentions a bucket lock/lease as a possible
approach.
- #266 / #183 — Cloud Run/serverless context.

So this is not a request to invent leasing from scratch. It is a request to expose
and integrate the existing leasing work into the replication lifecycle.

## Use Cases

On Cloud Run, Kubernetes, Kamal-style Docker deploys, and other
container/serverless schedulers, it is easy to briefly have two app instances
alive:

- a new instance starts before the old one is fully terminated;
- a zero-downtime rollout overlaps old and new containers;
- an instance crashes without clean shutdown and a replacement starts;
- a platform retries or reconciles a deploy step.

For apps using SQLite + Litestream, this can mean two Litestream processes
replicating to the same object-store path. The docs correctly warn that this can
corrupt the replica / make restore impossible, but each deployment environment
currently has to provide its own coordination.

The end goal is a simple config-level way to say:

```text
this Litestream process may write to the configured replica only while it holds
the configured lease
```

and for `replicate -exec` to terminate the child process if that invariant is
lost.

## Attempted Solutions

The workaround is to run a second supervisor above Litestream:

```text
external supervisor
-> acquire/heartbeat/release lease
-> run litestream restore
-> run litestream replicate -exec "my-app"
```

That feels like the wrong layering because `replicate -exec` already makes
Litestream the supervisor of the application process that uses the SQLite DB.
Litestream already owns the lifecycle boundary that matters:

```text
restore/replicate SQLite -> start app -> stop app on shutdown
```

The only missing part is making the existing lease a first-class precondition for
that lifecycle. Without that, every platform integration has to duplicate
conditional writes, TTLs, stale takeover, graceful shutdown, and child-process
termination around Litestream.

## Proposal

One possible minimal config shape:

```yaml
dbs:
- path: /data/app.db

lease:
required: true # default: false
ttl: 60s
heartbeat: 15s
acquire-timeout: 0s # 0 = wait forever

replica:
type: gcs
bucket: my-bucket
path: litestream/app.db
```

If `lease.required` is false or omitted, behavior is unchanged.

When `lease.required: true`, Litestream would:

1. acquire the lease before writing/replicating to the replica;
2. heartbeat while replication is active;
3. wait up to `acquire-timeout` to acquire the lease (`0s` = wait forever);
4. stop replication and terminate the `-exec` child if the lease is lost or
cannot be refreshed within `ttl`;
5. release the lease on graceful shutdown.

This should apply especially to:

```sh
litestream replicate -exec "my-app"
```

That would let app authors keep the normal Litestream entrypoint:

```sh
litestream restore -if-replica-exists -config /app/litestream.yml /data/app.db

exec litestream replicate \
-config /app/litestream.yml \
-exec "my-app"
```

while making accidental concurrent replication fail safe instead of corrupting
the restore chain.

I saw in #922 that Ben suggested a globally scoped lock ("any collection of
SQLite databases managed by a process should be locked together"). If that is
still the intended model, I am happy to adapt the config shape. The core ask is
not the exact YAML; it is that `replicate` can require and maintain a lease before
it writes, and can stop its `-exec` child if the lease is lost.

Non-goals:

- no multi-writer SQLite support;
- no platform-specific rollout orchestration;
- no HTTP readiness/proxy behavior;
- no behavior change unless `lease.required: true`.

If this direction is desirable, I would be happy to help with a focused PR. A
reasonable first PR could be small and scoped to:

- wire the existing #1073 S3 leaser into `replicate` / `replicate -exec`;
- add the minimal config fields;
- fail closed / terminate the `-exec` child if the lease is lost;
- include tests for lease acquisition, renewal/loss, and child termination.

Contributor guide

Open the contributing guide

Research direction

Start by reading the distributed leasing infrastructure from #1073 and the existing replicate and replicate -exec lifecycle. Determine how lease configuration should fit litestream.yml, including the unresolved process-wide versus per-database scope. Add focused tests for acquisition, renewal and loss, including termination of the -exec child, and verify the default behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.