apache / apache/pinot

Auto tune table config

Open
#6,254 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Make it easier for people to create a Pinot table which can execute queries in a performant way. Usually this means the Pinot user has to configure things such as inverted indexes. We can do this automatically.

For instance, users can specify something like:

```
"tableIndexConfig": {
“mode”: “auto”
"streamConfigs" : {
"streamType": "kafka",
...
}
}
```

which can be automatically replaced with the necessary config.

The implementation can be done in 2 parts:
1) Have a pluggable interface for defining such customizations
2) Iterate on different types of such auto-tuning implementations.

Contributor guide

Open the contributing guide

Research direction

Start by locating where Pinot parses and applies tableIndexConfig, then review how table configuration and streamConfigs are represented. The issue is complete when a pluggable customization interface is defined and an initial auto-tuning implementation can replace an auto mode with the necessary configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.