apache / apache/pinot

Backfill for partial upsert tables - need suggestions

Open
#12,331 1 comment 0 reactions 0 assignees View on GitHub
feature stale upsert
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

There are ways to upload pre-generated segments to Pinot tables. This provides a mechanism for backfills for realtime and offline tables (bootstrapping a new table or correcting data). If upsert metadata is properly provided for full upsert tables, then pre-partitioned segments by the primary key can also be uploaded with correct upsert behaviour.

The above approach falls short when uploading segments of partial upsert table as there is an insert time transformation on the row by merging it with previously persisted row for a primary key.

Instead of uploading pre-generated segments to partial upsert table, one can also not put the backfill data in the stream for realtime ingestion to take care of insert time transformation (merging) as during backfills one would want to remove all the rows of a primary key persisted and then have a fresh ingestion for the primary key.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.