tursodatabase / tursodatabase/libsql
SQLD replica is super slow (due to blocking)?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 17.2k
- Forks
- 531
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 1
Description
I've got a test setup of LibSQL SQLD running with a primary and a replica in a docker image. The docker-compose.yml is as follows:
version: "3"
services:
lsql-1:
image: ghcr.io/tursodatabase/libsql-server:latest
platform: linux/amd64
container_name: lsql-1
restart: no
ports:
- "8051:8051"
expose:
- 5001
volumes:
- ./data/1:/var/lib/sqld
environment:
- SQLD_NODE=primary
- SQLD_HTTP_AUTH=basic:YWRtaW46YWRtaW4=
- SQLD_GRPC_LISTEN_ADDR=0.0.0.0:5001
- SQLD_HTTP_LISTEN_ADDR=0.0.0.0:8051
lsql-2:
image: ghcr.io/tursodatabase/libsql-server:latest
platform: linux/amd64
container_name: lsql-2
restart: no
ports:
- "8052:8052"
expose:
- 5002
volumes:
- ./data/2:/var/lib/sqld
environment:
- SQLD_NODE=replica
- SQLD_HTTP_AUTH=basic:YWRtaW46YWRtaW4=
- SQLD_PRIMARY_URL=http://lsql-1:5001/
- SQLD_GRPC_LISTEN_ADDR=0.0.0.0:5002
- SQLD_HTTP_LISTEN_ADDR=0.0.0.0:8052
networks:
lsql:
driver: bridge
This is running on a Minisforum EM780 with 32 GB of RAM. Specs can be found here: https://store.minisforum.com/products/minisforum-em680
Using a test script that performs a number of basic read/write operations (visible here: https://github.com/hiraeth-php/turso/blob/master/test/index.php) I get wildly disparate execution times running it against the primary vs. the replica. The latter being an order of magnitude (over 1 second), slower. Here's the results from time runs:
Running on the Primary
________________________________________________________
Executed in 75.91 millis fish external
usr time 26.64 millis 1.46 millis 25.17 millis
sys time 15.02 millis 0.63 millis 14.38 millis
Running on the Replica
________________________________________________________
Executed in 1.16 secs fish external
usr time 39.42 millis 833.00 micros 38.59 millis
sys time 10.52 millis 0.00 micros 10.52 millis
You can see the replica is over 1 second, however its usr and sys times are not wildly different, which suggests the replica is doing a lot of blocking waiting on and/or syncing with the primary. None of this should really be network latency, as both instances are just two docker containers on the local system talking to one another.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported timings from docker-compose.yml using the test script in test/index.php, comparing the primary and replica containers. Start by checking replica synchronization and blocking behavior, then document the cause and a verified way to eliminate or explain the over-one-second delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, php
- Domain
- databases, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100