[CORE][VL] V2 columnar write
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
The topic is to enable DSV2 write in Gluten, also in preparation for Iceberg support.
This will involve:
1. Have a V2 columnar write API
2. Test: Add columnar version of `InMemoryTableCatalog` (e.g., `ColumnarInMemoryTableCatalog`) that fits into V2 columnar write API, but not necessarily do columnar processing internally (could reuse `InMemoryTableCatalog`'s row-based storage after the data passes through the columnar API)
3. Enable `DataFrameWriterV2Suite`
Contributor guide
Research direction
Start by locating the V2 write API and DataFrameWriterV2Suite, then inspect InMemoryTableCatalog to determine how a columnar test catalog would fit. Done means providing a V2 columnar write API, adding the columnar InMemoryTableCatalog test coverage, and enabling DataFrameWriterV2Suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend-api-design, data, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100