[Improvement] Push Hive partition filters to HMS during planning
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Problem
For highly partitioned Hive tables, FE planning may enumerate every HMS partition name before applying selective partition predicates. This makes planning latency depend on a full metastore listing and can cause timeouts.
## Proposal
Materialize safe equality and IN partition predicates through HMS `get_partitions_by_filter` before the generic selected-partition map is built. Keep the existing local full-list pruning path as a compatibility fallback when the HMS filter API or dialect is unavailable.
## Expected benefit
Selective partition queries avoid the full partition-name enumeration on the planner path when HMS supports the filter API.
Contributor guide
Research direction
Start in the FE planning path that builds the selected-partition map, and trace how Hive Metastore partition names are currently listed and pruned. Compare that flow with HMS get_partitions_by_filter for safe equality and IN predicates, while preserving the local full-list fallback when the API or dialect is unavailable. Done means selective queries avoid full enumeration when supported and retain compatibility otherwise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100