apache / apache/doris

[Improvement] Push Hive partition filters to HMS during planning

Open
#67,724 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Problem

For highly partitioned Hive tables, FE planning may enumerate every HMS partition name before applying selective partition predicates. This makes planning latency depend on a full metastore listing and can cause timeouts.

## Proposal

Materialize safe equality and IN partition predicates through HMS `get_partitions_by_filter` before the generic selected-partition map is built. Keep the existing local full-list pruning path as a compatibility fallback when the HMS filter API or dialect is unavailable.

## Expected benefit

Selective partition queries avoid the full partition-name enumeration on the planner path when HMS supports the filter API.

Contributor guide

Open the contributing guide

Research direction

Start in the FE planning path that builds the selected-partition map, and trace how Hive Metastore partition names are currently listed and pruned. Compare that flow with HMS get_partitions_by_filter for safe equality and IN predicates, while preserving the local full-list fallback when the API or dialect is unavailable. Done means selective queries avoid full enumeration when supported and retain compatibility otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.