[Feature] Create integrations with different storage engines with the Da Vinci Record Transformer (DVRT)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 611
- Forks
- 124
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 26
Description
### Willingness to contribute
No. I cannot contribute at this time.
### Feature Request Proposal
The [Da Vinci Record Transformer](https://venicedb.org/docs/user_guide/read_api/da_vinci_client#record-transformer) is an API that allows users to hook into the lifecycle of the Da Vinci Client.
This feature was built by the open-source community to give more leverage to our users. To showcase the power of DVRT, we would like to build integrations with it that can used by others or be used an example of what can be done with it.
One example integration done with DVRT was with [DuckDB](https://github.com/linkedin/venice/blob/main/integrations/venice-duckdb/src/main/java/com/linkedin/venice/duckdb/DuckDBDaVinciRecordTransformer.java). This allows users to run [SQL OLAP](https://support.microsoft.com/en-us/office/overview-of-online-analytical-processing-olap-15d2cdde-f70b-4277-b009-ed732b75fdd6#:~:text=Online%20Analytical%20Processing%20(OLAP)%20is,databases%20and%20support%20business%20intelligence.) queries against Venice datasets, which was not previously possible due to Venice's [key-value access pattern](https://www.mongodb.com/resources/basics/databases/key-value-database).
### Motivation
Since this is a brand new API that is directly exposed to users, we would like to showcase what is possible with it to encourage our users to onboard to it. Doing this exercise also helps identify any gaps in the DVRT abstract class.
### Details
Integrations will be built on top the [DaVinciRecordTransformer](https://github.com/linkedin/venice/blob/main/clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciRecordTransformer.java) abstract class. Please use the [DuckDB integration](https://github.com/linkedin/venice/blob/main/integrations/venice-duckdb/src/main/java/com/linkedin/venice/duckdb/DuckDBDaVinciRecordTransformer.java) as a frame of reference when developing.
Since Venice is a key-value database and DuckDB is a SQL OLAP database, we would like to have integrations with different types of databases. Some examples would be: graph databases or search engines. Please keep in mind performance and community usage when selecting a database to integrate with.
### What component(s) does this affect?
- [ ] `Controller`: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
- [ ] `Router`: This is the stateless query-routing layer for serving read requests.
- [ ] `Server`: This is the component that persists all the store data.
- [ ] `VenicePushJob`: This is the component that pushes derived data from Hadoop to Venice backend.
- [ ] `VenicePulsarSink`: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
- [ ] `Thin Client`: This is a stateless client users use to query Venice Router for reading store data.
- [ ] `Fast Client`: This is a stateful client users use to query Venice Server for reading store data.
- [x] `Da Vinci Client`: This is an embedded, stateful client that materializes store data locally.
- [ ] `Samza`: This is the library users use to make nearline updates to store data.
- [ ] `Admin Tool`: This is the stand-alone client used for ad-hoc operations on Venice.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciRecordTransformer.java and the existing integrations/venice-duckdb/src/main/java/com/linkedin/venice/duckdb/DuckDBDaVinciRecordTransformer.java. Select a storage engine or search integration based on performance and community usage, then use the DuckDB integration as the reference for defining and validating the new integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100