apache / apache/gluten

[CH] Support Delta Liquid Clustering on Mergetree

Open
#9,070 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

- [ ] Syntax support on spark3.5 [SPARK-48761](https://issues.apache.org/jira/browse/SPARK-48761)
- [ ] Dynamic cluster key pruning like DPP
- [x] Delta writes stats
- https://github.com/apache/incubator-gluten/pull/9047
- https://github.com/apache/incubator-gluten/pull/9089
- [ ] Delta write properties support (like dataSkippingNumIndexedCols) [DeltaDataSkipping](https://docs.databricks.com/aws/en/delta/data-skipping)
- [ ] Delta optimizes write with local clustering sort
- [ ] Delta liquid table metadata check
- [ ] Support Delta liquid clustering expression
- [ ] Delta log operation performance

Contributor guide

Open the contributing guide

Research direction

Start with the unchecked items in this issue, especially SPARK-48761, the DeltaDataSkipping reference, and the linked write-statistics pull requests. Determine which clustering, metadata, write-property, pruning, expression, and Delta-log performance requirements apply to Mergetree; done requires a defined implementation and validation for the supported checklist items.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.