[CH] Fully Support writing parquet and mergetree in spark 3.5.x with delta protocol
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
This is umbrella issue.
Previously, https://github.com/apache/incubator-gluten/issues/6705 is just a POC to prove that we can implemtent Delta Write based on `CumnarWriteFilesExec`.
1. https://github.com/apache/incubator-gluten/pull/7029
2. https://github.com/apache/incubator-gluten/pull/7279
3. https://github.com/apache/incubator-gluten/pull/7234
4. https://github.com/apache/incubator-gluten/pull/7170
5. https://github.com/apache/incubator-gluten/pull/7395
6. https://github.com/apache/incubator-gluten/pull/7506
7. https://github.com/apache/incubator-gluten/pull/7788
8. https://github.com/apache/incubator-gluten/pull/7924
9. https://github.com/apache/incubator-gluten/pull/7993
10. https://github.com/apache/incubator-gluten/pull/8029
11. https://github.com/apache/incubator-gluten/pull/8052
12. https://github.com/apache/incubator-gluten/pull/8237
13. https://github.com/apache/incubator-gluten/pull/8840 ~~https://github.com/apache/incubator-gluten/pull/8158~~
14. https://github.com/apache/incubator-gluten/pull/8346
15. https://github.com/apache/incubator-gluten/pull/8364
## backlog
- [x] Collect file bytes that write pipeline produces.
- [ ] https://github.com/apache/incubator-gluten/issues/8053
- [ ] Support `InsertIntoHiveTable`
- [ ] Support Bucket table? or drop support
- [x] ~`HDFS, HDFS with rockdb, S3` :: **test mergetree write with the path based bucket table**~ by Design => (#8364)
- [ ] Support escape partition value. e.g. incluing space #8158
- [x] we **need redesign** how to support case (in)sensitive (#8346).
- [x] Bug Fix (#8364)
- [x] test mergetree with partition with whitespace
- [x] `GlutenClickHouseMergeTreeCacheDataSuite::test cache mergetree data no partition columns`
- [x] `GlutenClickHouseMergeTreePathBasedWriteSuite::test mergetree path based table update` and `GlutenClickHouseMergeTreePathBasedWriteSuite::test mergetree path based table delete`
Contributor guide
Research direction
Start by reviewing the remaining unchecked backlog items, especially issue #8053, InsertIntoHiveTable, bucket-table support, and escaped partition values. Use the linked pull requests and the named GlutenClickHouseMergeTreePathBasedWriteSuite and GlutenClickHouseMergeTreeCacheDataSuite tests to understand what remains; done means the outstanding Spark 3.5.x parquet and MergeTree write cases are supported and covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100