apache / apache/pinot

Extend the partition segment pruner over a `composite` primary key

Open
#10,039 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

The primary key for upsert feature can be composite (https://docs.pinot.apache.org/basics/data-import/upsert#define-the-primary-key-in-the-schema).

This means that the user will partition data on `hash(colA, colB) when is a composite primary key`. Pinot's current custom partition based segment assignment & pruning can only be configured per column basis. There's no way to specify the partition based on a composite key.

This means that if the upsert is being used and the primary key is defined as composite, there's no way for us to prune the segments on the broker level and this would cause all queries to be routed to all servers. However, we can answer the query by hitting 1 server when there's a filter on the composite primary key.

We can optimize the routing side if the custom partitioning on a composite key is supported on the Pinot side.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pinot's custom partition-based segment assignment and broker-level segment pruning, then review how composite upsert primary keys are represented. Reproduce the case where hashing colA and colB prevents pruning, and define completion as routing a query filtered on the composite key to the appropriate server rather than all servers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.