Support for Materialised Query Rewrite in Apache Pinot using MV Tables and Calcite
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Currently, Apache Pinot supports star-tree indexes for pre-aggregation, but sometimes creating star-trees for wide-column tables is not effective due to excessive disk space consumption. To address this, introduce support for Materialised Views (MV) in Pinot:
**MV Creation:** Allow materialised view tables to be created externally (e.g., in Hive) and ingested into Pinot as MV tables.
**Metadata Mapping:** Maintain metadata linking MV tables to the main table, including the original query used for MV creation.
**Query Rewrite:** Integrate Calcite-based query rewriting to automatically select the most cost-effective MV table when serving queries.
**Further Optimisation:** Enable star-tree indexes on MV tables for frequently accessed aggregations. Since MV tables are smaller, star-trees will consume less space, improving performance.
**Goal:** Achieve multi-level aggregation and efficient real-time analytics for wide-column tables without prohibitive storage costs.
Pros
- Significant performance boost for ad-hoc queries and complex aggregations.
- Reduced disk space usage compared to star-trees on wide-column tables.
- Flexible design leveraging existing MV creation tools like Hive.
Cons
- Users must refresh MV tables when underlying data changes.
- Best suited for offline tables rather than real-time ingestion scenarios.
Contributor guide
Research direction
Start by reading Apache Pinot's existing star-tree and table-metadata entry points, then trace how Calcite is integrated into query planning. Investigate how externally created MV tables and their source queries could be represented and refreshed. Done means a decided design and working metadata, rewrite, and optimisation behavior for the stated offline-table scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100