apache / apache/pinot

[Time series Gap-fill V2] Server side gap-filling of time series data

Open
#9,492 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We recently introduced gapfill() to interpolate and fill gaps in a time series dataset. In the current solution, the data is moved and gapfilling is performed in the broker. This puts lots of stress on the broker as all of the raw records to be transferred from server to broker.

This brings in a lot of limitations and doesn't work for large dataset with larger date range. In V2, let us revisit the implementation to push down the gapfill and aggregations to server side by leveraging data locality and make this feature scale better.

cc @weixiangsun @Jackie-Jiang

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing gapfill() implementation and tracing how raw records and aggregations move through the broker. Then identify the server-side query path and the tests covering gap-filling. Done means gap-filling and aggregations execute server-side, reduce broker data transfer, and work for large datasets and date ranges.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.