apache / apache/gluten

[VL] Track the limitations and failed unit test list about spark 3.4 native parquet write.

Open
#4,110 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

**Limitations:**

1. Timestamp type fallback
2. Complex type fallback
3. Columns information with metadata are not currently supported.
4. Compression codecs like LZO and Brotli are not supported.
5. Partition key type restrictions: The supported partition key types are BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, VARCHAR, and VARBINARY.
6. Only spark compression codec config is supported.

**Failed unit test list:**

GlutenParquetFilterSuite

- [ ] GlutenTestConstants.GLUTEN_TEST + "Support Parquet column index"

GlutenSQLQuerySuite

- [x] GlutenTestConstants.GLUTEN_TEST + "SPARK-33338: GROUP BY using literal map should not fail"

GlutenSubquerySuite

- [x] "SPARK-42745: Improved AliasAwareOutputExpression works with DSv2"

GlutenStatisticsCollectionSuite

- [x] column stats collection for null columns
- [x] store and retrieve column stats in different time zones
- [x] "SPARK-38140: describe column stats (min, max) for timestamp column: desc results should be consistent with the written value if writing and desc happen in the same time zone")
- [x] "SPARK-38140: describe column stats (min, max) for timestamp column: desc should show different results if writing in UTC and desc in other time zones
- [x] Gluten - store and retrieve column stats in different time zones

GlutenSQLQuerySuite

- [x] SPARK-26709: OptimizeMetadataOnlyQuery does not handle empty records correctly

GlutenFileSourceSQLInsertTestSuite

- [x] SPARK-30844: static partition should also follow StoreAssignmentPolicy

GlutenFileBasedDataSourceSuite

- [x] SPARK-15474 Write and read back non-empty schema with empty dataframe
- [x] SPARK-23271 empty RDD when saved should write a metadata only file

GlutenPartitionedWriteSuite

- [x] maxRecordsPerFile setting in non-partitioned write path

GlutenInsertSuite

- [x] SPARK-35106: Throw exception when rename custom partition paths returns false
- [ ] Stop task set if FileAlreadyExistsException was thrown

GlutenFileFormatWriterSuite

- [x] empty file should be skipped while write to file

GlutenParquetRebaseDatetimeV2Suite

- [ ] SPARK-33163, SPARK-37705: write the metadata keys

GlutenParquetRebaseDatetimeV1Suite

- [ ] SPARK-33163, SPARK-37705: write the metadata keys

GlutenParquetV2QuerySuite

- [x] SPARK-11997 parquet with null partition values

GlutenParquetV1QuerySuite

- [x] SPARK-11997 parquet with null partition values

GlutenParquetIOSuite

- [ ] Write Spark version into Parquet metadata
- [x] SPARK-7837 Do not close output writer twice when commitTask() fails

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.