tursodatabase / tursodatabase/libsql
Feature request: support for `begin concurrent`
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 17.2k
- Forks
- 531
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 1
Description
We are using the begin concurrent branch (https://www.sqlite.org/src/vdiff?branch=begin-concurrent -- see also https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md) of sqlite for https://zerosync.dev.
We use it in kind of an interesting way: we don't ever commit the concurrent transactions. Instead, we open concurrent write transactions, read and write data to them, then rollback the transaction.
The reason we want to do this is because to accomplish its work, zero needs to maintain materialized views of queries from the client. While cheaper than running each client query from scratch, maintaining these views can still be expensive so we want to do it on multiple threads.
The algorithm we have developed to maintain these views requires a series of interactive write/read steps. For an incoming write transaction, we write a row to the datastore (sqlite), do some queries, write another row, do some queries, etc. At the end of these series of steps we have the output that we send to the client. We cannot do all the writes up front - the correctness of the algorithm requires the write/read interleaving.
If we run this algorithm on multiple thread in parallel, the writes will take eager locks and conflict with each other. But begin concurrent give us an awesome out: we can open transactions in parallel, do our write/read pairs, then abort the transactions. Then we just have a single 'writer' transaction that actually flushes the persistent writes to the db at the end.
Unfortunately we have to use this kind of semi-zombie branch of sqlite to accomplish this. It would be super useful to have it officially supported in libsql.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked begin-concurrent documentation and comparing the begin-concurrent SQLite branch with libSQL's current transaction support. Trace the transaction entry points involved in opening, interleaving, and rolling back concurrent write transactions. Done means libSQL officially supports the requested begin concurrent workflow for the described parallel read/write transactions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, sqlite
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100