benbjohnson / benbjohnson/litestream
feat: Expose distributed leasing in replicate -exec lifecycle
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 417
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
## Feature Description
Expose Litestream's distributed leasing support in `litestream.yml` and enforce it
inside `litestream replicate` / `replicate -exec`, so users can opt into a
fail-safe guard against multiple Litestream processes writing to the same replica
location.
Litestream already documents the caveat this would address:
> Multiple applications replicating into the same bucket & path can cause
> situations where you will be unable to restore. It is your responsibility to
> ensure you do not have multiple applications replicating concurrently.
>
> https://litestream.io/tips/#multiple-applications-replicating-into-location-can-corrupt
I searched related issues/PRs and found:
- #1073 — merged S3 distributed leasing infrastructure.
- #922 — earlier leasing design discussion, closed in favor of #1073.
- #1119 — Docker zero-downtime deploy question; the maintainer response says
leasing infrastructure exists but is not yet integrated into the replication
loop.
- #224 — restore failure caused by another machine writing to the same bucket
location; the discussion explicitly mentions a bucket lock/lease as a possible
approach.
- #266 / #183 — Cloud Run/serverless context.
So this is not a request to invent leasing from scratch. It is a request to expose
and integrate the existing leasing work into the replication lifecycle.
## Use Cases
On Cloud Run, Kubernetes, Kamal-style Docker deploys, and other
container/serverless schedulers, it is easy to briefly have two app instances
alive:
- a new instance starts before the old one is fully terminated;
- a zero-downtime rollout overlaps old and new containers;
- an instance crashes without clean shutdown and a replacement starts;
- a platform retries or reconciles a deploy step.
For apps using SQLite + Litestream, this can mean two Litestream processes
replicating to the same object-store path. The docs correctly warn that this can
corrupt the replica / make restore impossible, but each deployment environment
currently has to provide its own coordination.
The end goal is a simple config-level way to say:
```text
this Litestream process may write to the configured replica only while it holds
the configured lease
```
and for `replicate -exec` to terminate the child process if that invariant is
lost.
## Attempted Solutions
The workaround is to run a second supervisor above Litestream:
```text
external supervisor
-> acquire/heartbeat/release lease
-> run litestream restore
-> run litestream replicate -exec "my-app"
```
That feels like the wrong layering because `replicate -exec` already makes
Litestream the supervisor of the application process that uses the SQLite DB.
Litestream already owns the lifecycle boundary that matters:
```text
restore/replicate SQLite -> start app -> stop app on shutdown
```
The only missing part is making the existing lease a first-class precondition for
that lifecycle. Without that, every platform integration has to duplicate
conditional writes, TTLs, stale takeover, graceful shutdown, and child-process
termination around Litestream.
## Proposal
One possible minimal config shape:
```yaml
dbs:
- path: /data/app.db
lease:
required: true # default: false
ttl: 60s
heartbeat: 15s
acquire-timeout: 0s # 0 = wait forever
replica:
type: gcs
bucket: my-bucket
path: litestream/app.db
```
If `lease.required` is false or omitted, behavior is unchanged.
When `lease.required: true`, Litestream would:
1. acquire the lease before writing/replicating to the replica;
2. heartbeat while replication is active;
3. wait up to `acquire-timeout` to acquire the lease (`0s` = wait forever);
4. stop replication and terminate the `-exec` child if the lease is lost or
cannot be refreshed within `ttl`;
5. release the lease on graceful shutdown.
This should apply especially to:
```sh
litestream replicate -exec "my-app"
```
That would let app authors keep the normal Litestream entrypoint:
```sh
litestream restore -if-replica-exists -config /app/litestream.yml /data/app.db
exec litestream replicate \
-config /app/litestream.yml \
-exec "my-app"
```
while making accidental concurrent replication fail safe instead of corrupting
the restore chain.
I saw in #922 that Ben suggested a globally scoped lock ("any collection of
SQLite databases managed by a process should be locked together"). If that is
still the intended model, I am happy to adapt the config shape. The core ask is
not the exact YAML; it is that `replicate` can require and maintain a lease before
it writes, and can stop its `-exec` child if the lease is lost.
Non-goals:
- no multi-writer SQLite support;
- no platform-specific rollout orchestration;
- no HTTP readiness/proxy behavior;
- no behavior change unless `lease.required: true`.
If this direction is desirable, I would be happy to help with a focused PR. A
reasonable first PR could be small and scoped to:
- wire the existing #1073 S3 leaser into `replicate` / `replicate -exec`;
- add the minimal config fields;
- fail closed / terminate the `-exec` child if the lease is lost;
- include tests for lease acquisition, renewal/loss, and child termination.
Contributor guide
Research direction
Start by reading the distributed leasing infrastructure from #1073 and the existing replicate and replicate -exec lifecycle. Determine how lease configuration should fit litestream.yml, including the unresolved process-wide versus per-database scope. Add focused tests for acquisition, renewal and loss, including termination of the -exec child, and verify the default behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100