bytecodealliance / bytecodealliance/wasmtime

Compiling SQL queries with Cranelift?

Open
#12,238 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
18.6k
Forks
1.8k
Avg merge
1d 19h
Merged PRs (30d)
121

Description

I'm building a next generation distributed datastore and I would like to add the possibility of querying using a relational model. A few weeks ago I had the crazy idea of compiling SQL prepared statements instead of running them through a traditional query planner, before finding out I'm not the first one to use this approach.

Right now I _transliterate_ the query to Rust before compiling it, but maybe there is a smarter approach. My questions are:

- Can Cranelift support this use case or is it too far from its original design?
- Could Cranelift help me skip the _transliteration_ step?
- Are you aware of anyone else using this same approach with Cranelift instead of LLVM?

**More details:**
Let's say I have a query in the form:

```
SELECT
ot.customer_ID,
ct.customer_name,
SUM(ot.order_total) as total_amount,
UDF.heavy_function(ot.order_total) AS processed_total
FROM orders_table AS ot
JOIN
customers_table AS ct ON ct.customer_ID = ot.customer_ID
WHERE
ot.customer_ID IN $customers_array
AND
ot.order_date BETWEEN $start_date AND $end_date
GROUP BY
ot.customer_ID,
ct.customer_name;
```

Right now the compilation looks like:
- The user register the SQL query before executing it
- I parse the query using an SQL parser
- I extract the parameters, these will enter in the signature of the Rust function representing the query
- I _transliterate_ to Rust using some very ugly code (a bunch of `match`, `if` and generic types)
- I run some compile time optimizations (using a cost based function, like the vulcano query planner)
- I compile the code
- I cache the compiled output
- I notify the user that the query is ready to use

To run the query:
- The user calls the query and provide the inputs
- I search for the cached compiled output using an hash of the query
- I launch one thread for each shard
- I reduce the results and present it to the user

Ideally I would like to run the query inside a WASM runtime.

I apologize in advance if this question is off topic for this repo and for my noobishness.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.