apache / apache/gluten

[CH] Fully Support writing parquet and mergetree in spark 3.5.x with delta protocol

Open
#7,028 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

This is umbrella issue.

Previously, https://github.com/apache/incubator-gluten/issues/6705 is just a POC to prove that we can implemtent Delta Write based on `CumnarWriteFilesExec`.

1. https://github.com/apache/incubator-gluten/pull/7029
2. https://github.com/apache/incubator-gluten/pull/7279
3. https://github.com/apache/incubator-gluten/pull/7234
4. https://github.com/apache/incubator-gluten/pull/7170
5. https://github.com/apache/incubator-gluten/pull/7395
6. https://github.com/apache/incubator-gluten/pull/7506
7. https://github.com/apache/incubator-gluten/pull/7788
8. https://github.com/apache/incubator-gluten/pull/7924
9. https://github.com/apache/incubator-gluten/pull/7993
10. https://github.com/apache/incubator-gluten/pull/8029
11. https://github.com/apache/incubator-gluten/pull/8052
12. https://github.com/apache/incubator-gluten/pull/8237
13. https://github.com/apache/incubator-gluten/pull/8840 ~~https://github.com/apache/incubator-gluten/pull/8158~~
14. https://github.com/apache/incubator-gluten/pull/8346
15. https://github.com/apache/incubator-gluten/pull/8364

## backlog
- [x] Collect file bytes that write pipeline produces.
- [ ] https://github.com/apache/incubator-gluten/issues/8053
- [ ] Support `InsertIntoHiveTable`
- [ ] Support Bucket table? or drop support
- [x] ~`HDFS, HDFS with rockdb, S3` :: **test mergetree write with the path based bucket table**~ by Design => (#8364)
- [ ] Support escape partition value. e.g. incluing space #8158
- [x] we **need redesign** how to support case (in)sensitive (#8346).
- [x] Bug Fix (#8364)
- [x] test mergetree with partition with whitespace
- [x] `GlutenClickHouseMergeTreeCacheDataSuite::test cache mergetree data no partition columns`
- [x] `GlutenClickHouseMergeTreePathBasedWriteSuite::test mergetree path based table update` and `GlutenClickHouseMergeTreePathBasedWriteSuite::test mergetree path based table delete`

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the remaining unchecked backlog items, especially issue #8053, InsertIntoHiveTable, bucket-table support, and escaped partition values. Use the linked pull requests and the named GlutenClickHouseMergeTreePathBasedWriteSuite and GlutenClickHouseMergeTreeCacheDataSuite tests to understand what remains; done means the outstanding Spark 3.5.x parquet and MergeTree write cases are supported and covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.