tursodatabase / tursodatabase/libsql

Internal connection timeout with low concurrency

Open
#1,195 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug server
Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

I am able to trigger a internal connection timeout easily with even low concurrency on local development machine.

I ran cargo run --release in libsql-server directory and then generated some load with tursotest stress:

go run main.go stress http://localhost:8080 -c 10 -i 1000 -w insert
go run main.go stress http://localhost:8080 -c 10 -i 100000 -w select

which quickly starts to trigger the following:

Welcome to sqld!

version: 0.23.0
commit SHA: e388a72f2d05bd49b6e946d50f94c45f50f2e22e
build date: 2024-03-12

This software is in BETA version.
If you encounter any bug, please open an issue at https://github.com/tursodatabase/libsql/issues

config:
        - mode: standalone
        - database path: data.sqld
        - extensions path: <disabled>
        - listening for HTTP requests on: 127.0.0.1:8080
        - grpc_tls: no
2024-03-12T10:23:00.992876Z  INFO sqld: listening for incoming user HTTP connection on 127.0.0.1:8080
2024-03-12T10:23:01.001489Z  INFO new:restore: libsql_server::namespace::meta_store: restoring meta store
2024-03-12T10:23:01.001532Z  INFO new:restore: libsql_server::namespace::meta_store: meta store restore completed
2024-03-12T10:23:01.004357Z  INFO libsql_server: Server sending heartbeat to URL <not supplied> every 30s
2024-03-12T10:23:01.006199Z  INFO libsql_server::replication::primary::logger: SQLite autocheckpoint: 1000
2024-03-12T10:23:01.010916Z  INFO libsql_server::namespace::store: loaded namespace: `default`
2024-03-12T10:24:41.757702Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection
2024-03-12T10:24:41.757706Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection
2024-03-12T10:24:41.757742Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection
2024-03-12T10:24:41.757768Z ERROR libsql_server::error: HTTP API: 500 Internal Server Error, Anyhow(Timed out while opening database connection)
2024-03-12T10:24:41.757748Z ERROR libsql_server::error: HTTP API: 500 Internal Server Error, Anyhow(Timed out while opening database connection)
2024-03-12T10:24:41.757780Z ERROR libsql_server::error: HTTP API: 500 Internal Server Error, Anyhow(Timed out while opening database connection)
2024-03-12T10:24:41.953718Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection
2024-03-12T10:24:41.953739Z ERROR libsql_server::error: HTTP API: 500 Internal Server Error, Anyhow(Timed out while opening database connection)
2024-03-12T10:24:42.002774Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection
2024-03-12T10:24:42.002796Z ERROR libsql_server::error: HTTP API: 500 Internal Server Error, Anyhow(Timed out while opening database connection)
2024-03-12T10:24:42.083788Z ERROR libsql_server::hrana::http: hrana server: Timed out while opening database connection

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the timeout from the libsql-server directory with the shown tursotest stress commands and inspect the hrana HTTP connection-opening path, especially the libsql_server::hrana::http errors. Done means the same low-concurrency insert and select workloads no longer produce database-opening timeouts or HTTP 500 responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust, sqlite
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.