Real time server query performance hit when catching up Kafka backlogs
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Right now our upstream Kafka log has a "batching" behavior that every a few minutes there is a sudden spike in Kafka traffic (due to upstream Flink job's checkpointing behavior). This cause a periodical Kafka backlog for our Pinot server. For example, every 30 minutes, we will suddenly have lots of Kafka backlog to catch up.
This is fine but the query performance suffers a lot during that backlog period. Based on our quick chart analysis it could be I/O bound.
Below are some of the charts we can share, showing correlation.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by reviewing the three linked charts and the reported correlation between Kafka backlog and query performance, then identify the relevant Pinot ingestion and query paths; done should include a verified explanation of the I/O-bound behavior and a measured performance improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- databases, performance, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100