[CH] Support Delta Liquid Clustering on Mergetree
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
- [ ] Syntax support on spark3.5 [SPARK-48761](https://issues.apache.org/jira/browse/SPARK-48761)
- [ ] Dynamic cluster key pruning like DPP
- [x] Delta writes stats
- https://github.com/apache/incubator-gluten/pull/9047
- https://github.com/apache/incubator-gluten/pull/9089
- [ ] Delta write properties support (like dataSkippingNumIndexedCols) [DeltaDataSkipping](https://docs.databricks.com/aws/en/delta/data-skipping)
- [ ] Delta optimizes write with local clustering sort
- [ ] Delta liquid table metadata check
- [ ] Support Delta liquid clustering expression
- [ ] Delta log operation performance
Contributor guide
Research direction
Start with the unchecked items in this issue, especially SPARK-48761, the DeltaDataSkipping reference, and the linked write-statistics pull requests. Determine which clustering, metadata, write-property, pruning, expression, and Delta-log performance requirements apply to Mergetree; done requires a defined implementation and validation for the supported checklist items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100