apache / apache/pinot

Support for Materialised Query Rewrite in Apache Pinot using MV Tables and Calcite

Open
#17,298 5 comments 0 reactions 0 assignees View on GitHub
PEP-Request
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Currently, Apache Pinot supports star-tree indexes for pre-aggregation, but sometimes creating star-trees for wide-column tables is not effective due to excessive disk space consumption. To address this, introduce support for Materialised Views (MV) in Pinot:

**MV Creation:** Allow materialised view tables to be created externally (e.g., in Hive) and ingested into Pinot as MV tables.
**Metadata Mapping:** Maintain metadata linking MV tables to the main table, including the original query used for MV creation.
**Query Rewrite:** Integrate Calcite-based query rewriting to automatically select the most cost-effective MV table when serving queries.
**Further Optimisation:** Enable star-tree indexes on MV tables for frequently accessed aggregations. Since MV tables are smaller, star-trees will consume less space, improving performance.
**Goal:** Achieve multi-level aggregation and efficient real-time analytics for wide-column tables without prohibitive storage costs.

Pros

- Significant performance boost for ad-hoc queries and complex aggregations.
- Reduced disk space usage compared to star-trees on wide-column tables.
- Flexible design leveraging existing MV creation tools like Hive.

Cons

- Users must refresh MV tables when underlying data changes.
- Best suited for offline tables rather than real-time ingestion scenarios.

Contributor guide

Open the contributing guide

Research direction

Start by reading Apache Pinot's existing star-tree and table-metadata entry points, then trace how Calcite is integrated into query planning. Investigate how externally created MV tables and their source queries could be represented and refreshed. Done means a decided design and working metadata, rewrite, and optimisation behavior for the stated offline-table scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.