tursodatabase / tursodatabase/libsql

Bulk insertion, errors and unclear limits

Open
#1,280 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

In my project I'm inserting some 5000 rows of data in a transaction.

The first thing I bumped into was limitations in the query size. Queries that were too large (in my case larger than about 50 rows) resulted in the cryptic error:

LibsqlError: SERVER_ERROR: Server returned HTTP status 400: SQL text 1 not found

After figuring out that limitation I switched to a batching approach where I would insert smaller numbers of rows in a for loop inside a transaction.

This is when I bumped into a new error.

After about 5 seconds I would get the following error:
cannot rollback - no transaction is active

What's going on here? Will the queries in the transaction that were executed before this point be rolled back?

Following the suggestion of someone on Discord I tried running the same queries after updating my database from 0.24.2 to 0.24.4. This seems to have fixed this behaviour!

What are the limitations regarding query size and transaction duration?

Edit:
Removed the stuff about queries not being rolled back because I'm not 100% sure that happened, and I'm now unable to reproduce the behaviour even on my old database.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported bulk insertion and transaction behavior with about 5,000 rows on versions 0.24.2 and 0.24.4, checking query-size and transaction-duration limits. Done means the limits and the cause of the rollback error are documented or the differing behavior is fixed and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.