Backfill for partial upsert tables - need suggestions
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
There are ways to upload pre-generated segments to Pinot tables. This provides a mechanism for backfills for realtime and offline tables (bootstrapping a new table or correcting data). If upsert metadata is properly provided for full upsert tables, then pre-partitioned segments by the primary key can also be uploaded with correct upsert behaviour.
The above approach falls short when uploading segments of partial upsert table as there is an insert time transformation on the row by merging it with previously persisted row for a primary key.
Instead of uploading pre-generated segments to partial upsert table, one can also not put the backfill data in the stream for realtime ingestion to take care of insert time transformation (merging) as during backfills one would want to remove all the rows of a primary key persisted and then have a fresh ingestion for the primary key.
Contributor guide
Assessment
This issue has not been assessed yet.