InseeFr / InseeFr/Trevas

Support translation of VTL to/from Substrait to be able to run it on many different execution engines

Open
#377 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
20
Forks
7
Avg merge
1d 5h
Merged PRs (30d)
12

Description

Currently Trevas needs to implement a translation to each execution engine (e.g. Java or Spark) individually. This is time-consuming and requires possibly a lot of support and maintenance effort.

Trevas could in the future support a translation to [Substrait](https://substrait.io/). Substrait is a format for describing compute on structured data. It can be interpreted by any execution engine.

The [source code repository](https://github.com/substrait-io) and the ecosystem website [lists already a couple of supported engines](https://substrait.io/community/powered_by/):
* Acerco
* Arrow Flight SQL
* Datafusion
* DuckDB
* Java
* [Spark](https://github.com/substrait-io/substrait-java/tree/main/examples/substrait-spark)
* Python
* Javascript
* C++
* R/dplyr
* ...

This means by Trevas supporting Substrait VTL can be executed on single nodes (e.g. using Arrow) or on many nodes (e.g. Spark) while keeping the maintenance effort low.

Additionally, existing SQL queries in other engines can be translated to substrait and back to VTL to not rewrite everything from scratch.

While substrait is new it seems to support already a rich ecosystem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.