tursodatabase / tursodatabase/libsql

Libsql client: generalize data set access abstractions between local/replica/hrana connections

Open
#725 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

At the moment our libsql client is going to support several different providers:

  • Local SQLite connection
  • Embedded replica connections
  • Hrana HTTP connections - in two variants:
    • standard based on tokio/hyper
    • WebAssembly (with current Cloudflare API as a target)

The problems start when it comes to consolidating all of these variants into a single concise user-facing API.
At the moment this is realized as a wrapper around the following set of internal abstractions.

trait Conn {
    async fn execute(&self, sql: &str, params: Params) -> Result<u64>;
    async fn execute_batch(&self, sql: &str) -> Result<()>;
    async fn prepare(&self, sql: &str) -> Result<Statement>;
    async fn transaction(&self, tx_behavior: TransactionBehavior) -> Result<Transaction>;
    async fn is_autocommit(&self) -> Result<bool>;
    fn changes(&self) -> u64;
    fn last_insert_rowid(&self) -> i64;
}

trait Tx {
    async fn commit(&mut self) -> Result<()>;
    async fn rollback(&mut self) -> Result<()>;
}

trait Stmt {
    async fn execute(&mut self, params: &Params) -> Result<usize>;
    async fn query(&mut self, params: &Params) -> Result<Rows>;
    fn finalize(&mut self);
    fn reset(&mut self);
    fn parameter_count(&self) -> usize;
    fn parameter_name(&self, idx: i32) -> Option<&str>;
    fn columns(&self) -> Vec<Column>;
}

trait RowsInner {
    fn next(&mut self) -> Result<Option<Row>>;
    fn column_count(&self) -> i32;
    fn column_name(&self, idx: i32) -> Option<&str>;
    fn column_type(&self, idx: i32) -> Result<ValueType>;
}

trait RowInner {
    fn column_value(&self, idx: i32) -> Result<Value>;
    fn column_str(&self, idx: i32) -> Result<&str>;
    fn column_name(&self, idx: i32) -> Option<&str>;
    fn column_type(&self, idx: i32) -> Result<ValueType>;
    fn column_count(&self) -> usize;
}

These abstractions are modeled after current capabilities of SQLite FFI - which was good when our clients were mainly local SQLite and replica API (which also used very similar API). What this doesn't take into account is Hrana HTTP protocol capabilities:

  1. Hrana HTTP API v3 is capable or returning multiple result sets when calling Conn::execute_batch. These result sets are returned one by one using async streams. Atm. I don't see how we could generalize it over local/replica connections.
  2. Hrana HTTP API v3 is also capable of returning non-buffered rows - it's important to us, as SQL queries may result potentially big data sets, while the client is executed over low-provisioned machine. This means that RowsInner::next is also potentially an async stream of rows.

IMO current problem is that this API is modeled to almost match 1-1 SQLite FFI. It's not generalized enough. Additionally everything is also hidden behind proxy objects that are basically wrappers around Box<dyn RowsInner>/Box<dyn RowInner> etc.

What I could propose is to:

  1. Reduce API surface by consolidating column-related methods: if needed they can be evaluated lazily and reused. But most importantly, a definition of columns remains the same across Stmt/DataSet/Row. I'm not sure if keeping low level API of separate column_count/column_type/column_name has any advantage, but it requires Hrana to "downgrade" in order to satisfy this API.
  2. RowsInnerDataSet is implementing async stream row generation.
  3. Use associated types to provide a way to carry info about data sets and rows abstractions and only Box<dyn T> them at the highest level of the API (right now every connection provider has to boxify them on their own).
trait Conn {
    type BatchResult; // () for local/replication, futures::Stream<Item=Result<impl DataSet>> for Hrana
    async fn execute_batch(&self, sql: &str) -> Result<Self::BatchResult>; 
    // other methods are unchanged
}
trait Stmt {
    type DataSet: DataSet;

    async fn query(&mut self, params: &Params) -> Result<Self::DataSet>;
    fn columns(&self) -> &[Column];
    // other methods are unchanged
}
// A single data set with definition of rows (streamed) and columns.
trait DataSet : futures::Stream<Item=Result<Self::Row>> {
    type Row: Row;
    fn columns(&self) -> &[Column]; // columns could be lazily evaluated
}
// In scope of a single row all values are always eagerly evaluated. Therefore we can apply iterator
trait Row {
    type Iter: Iterator<Item=Value> + FixedSizeIterator;
    
    fn columns(&self) -> &[Columns];
    fn iter(&'a self) -> Self::Iter;
    fn get_value(&self, idx: i32) -> Option<Value>; // no need for result
}
struct Column {
    name: Option<String>,
    decl_type: Option<&str>,
}

This is to keep the API size minimal, as it needs to be reimplemented between local/replica/hrana connections (in last case potentially between buffered Hrana v2 and cursors in Hrana v3).

This should also allow to reuse more code in Cloudlfare context and potentially use specialisations that are not possible outside of hrana.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the current Conn, Tx, Stmt, RowsInner, and RowInner abstractions and their local, replica, and Hrana implementations. Compare them with the proposed Conn, Stmt, DataSet, and Row traits, then define completion as one consistent API that supports local and replica access alongside Hrana batch and streamed-row results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite, wasm
Domain
backend-api-design, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.