Framework to certify Pinot release (functional + performance)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
With the Pinot community growing, as well as Pinot evolving for the better, we anticipate a ton of features that would go into performance critical parts of the code (e.g. query execution, RT segment generation, etc). Given that Pinot's forte is its blazingly fast speed, we would like to ensure that the rapid feature development does not compromise on its performance and functionality. We need a framework using which any PR can be evaluated for performance/functionality. Some ideas:
Query Performance:
1. Create a few synthetic data sets with different characteristics that represent existing Pinot deployments across the community. We can probably enhance the existing data generator tool to server these requirements.
2. Generate a query set for a given data set to ensure desired level of selectivity.
3. Framework that runs a desired workload (qps) on these data sets with and without the changes from a PR, and reports the performance changes.
Realtime:
1. Consumption rate
2. Segment generation time
Ability to check on a variety of metrics, for example:
1. Latency p95/p99
2. Throughput
3. GC
https://docs.google.com/document/d/1QweQObrwGrcMR47dKzg1ZQpQkj4udHttkNm5kcMVBQ8/edit?usp=sharing
Contributor guide
Research direction
Start with the linked design document and inspect Pinot's existing data generator tool, which the issue identifies as a possible foundation. Define the framework scope from the listed query and realtime workloads and metrics, then confirm that it can compare PR changes for functionality, latency, throughput, GC, consumption rate, and segment generation time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100