[VL] Track the limitations and failed unit test list about spark 3.4 native parquet write.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
**Limitations:**
1. Timestamp type fallback
2. Complex type fallback
3. Columns information with metadata are not currently supported.
4. Compression codecs like LZO and Brotli are not supported.
5. Partition key type restrictions: The supported partition key types are BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, VARCHAR, and VARBINARY.
6. Only spark compression codec config is supported.
**Failed unit test list:**
GlutenParquetFilterSuite
- [ ] GlutenTestConstants.GLUTEN_TEST + "Support Parquet column index"
GlutenSQLQuerySuite
- [x] GlutenTestConstants.GLUTEN_TEST + "SPARK-33338: GROUP BY using literal map should not fail"
GlutenSubquerySuite
- [x] "SPARK-42745: Improved AliasAwareOutputExpression works with DSv2"
GlutenStatisticsCollectionSuite
- [x] column stats collection for null columns
- [x] store and retrieve column stats in different time zones
- [x] "SPARK-38140: describe column stats (min, max) for timestamp column: desc results should be consistent with the written value if writing and desc happen in the same time zone")
- [x] "SPARK-38140: describe column stats (min, max) for timestamp column: desc should show different results if writing in UTC and desc in other time zones
- [x] Gluten - store and retrieve column stats in different time zones
GlutenSQLQuerySuite
- [x] SPARK-26709: OptimizeMetadataOnlyQuery does not handle empty records correctly
GlutenFileSourceSQLInsertTestSuite
- [x] SPARK-30844: static partition should also follow StoreAssignmentPolicy
GlutenFileBasedDataSourceSuite
- [x] SPARK-15474 Write and read back non-empty schema with empty dataframe
- [x] SPARK-23271 empty RDD when saved should write a metadata only file
GlutenPartitionedWriteSuite
- [x] maxRecordsPerFile setting in non-partitioned write path
GlutenInsertSuite
- [x] SPARK-35106: Throw exception when rename custom partition paths returns false
- [ ] Stop task set if FileAlreadyExistsException was thrown
GlutenFileFormatWriterSuite
- [x] empty file should be skipped while write to file
GlutenParquetRebaseDatetimeV2Suite
- [ ] SPARK-33163, SPARK-37705: write the metadata keys
GlutenParquetRebaseDatetimeV1Suite
- [ ] SPARK-33163, SPARK-37705: write the metadata keys
GlutenParquetV2QuerySuite
- [x] SPARK-11997 parquet with null partition values
GlutenParquetV1QuerySuite
- [x] SPARK-11997 parquet with null partition values
GlutenParquetIOSuite
- [ ] Write Spark version into Parquet metadata
- [x] SPARK-7837 Do not close output writer twice when commitTask() fails
Contributor guide
Assessment
This issue has not been assessed yet.