Support translation of VTL to/from Substrait to be able to run it on many different execution engines
- Dominant language
- Java
- Stars
- 20
- Forks
- 7
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 12
Description
Currently Trevas needs to implement a translation to each execution engine (e.g. Java or Spark) individually. This is time-consuming and requires possibly a lot of support and maintenance effort.
Trevas could in the future support a translation to [Substrait](https://substrait.io/). Substrait is a format for describing compute on structured data. It can be interpreted by any execution engine.
The [source code repository](https://github.com/substrait-io) and the ecosystem website [lists already a couple of supported engines](https://substrait.io/community/powered_by/):
* Acerco
* Arrow Flight SQL
* Datafusion
* DuckDB
* Java
* [Spark](https://github.com/substrait-io/substrait-java/tree/main/examples/substrait-spark)
* Python
* Javascript
* C++
* R/dplyr
* ...
This means by Trevas supporting Substrait VTL can be executed on single nodes (e.g. using Arrow) or on many nodes (e.g. Spark) while keeping the maintenance effort low.
Additionally, existing SQL queries in other engines can be translated to substrait and back to VTL to not rewrite everything from scratch.
While substrait is new it seems to support already a rich ecosystem.
Contributor guide
Assessment
This issue has not been assessed yet.