apache / apache/pinot

[upsert] integrate partial upsert with groovy evaluator

Open
#11,174 4 comments 0 reactions 2 assignees Claimed by @rohityadav1993 View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Partial upsert let users choose to update only specific columns and ignore the rest, and apply complex merging logic based on the existing row and the new row that shares the same primary key.

Pinot allows you to run any function using [Apache Groovy](https://groovy-lang.org/) scripts. To support complex use cases (e.g merge previous and new values based on conditions), we proposed to enhance partial upsert with groovy transformation function.

a few challenges to consider

- 1. limitation of Groovy function regex, currently the groovy evaluator doesn't support multi-line groovy script.
- 2. FunctionEvaluator interface currently only take one GenericRow as input.
- 2.1 For partial upsert we need to access both old row and the new row.
- 2.2 How to represent the argument and how to bind the arguments with values for both previous row and the new row.
- 3. Integrate Groovy function in the partial upsert handler

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.