Spark 4.x: Tracking disabled test suites
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Please leave a comment and mark on the TODO tests in this issue if you want to contribute
## Backend
VL(Velox)
## Bug Description
These suites were commented out in https://github.com/apache/incubator-gluten/pull/11512 with TODO markers(and were updated based on #11800) indicating they need to be fixed and re-enabled.
## Context
The test suites below have been disabled due to various failures. Each table shows the status for both Spark 4.0 and 4.1:
- π΄ = Suite is commented out (disabled)
- π’ = Suite is enabled
The failure count (when available) is shown in the status column.
---
### org.apache.spark.sql
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenDataFrameSubquerySuite` | π’ | π’ | @baibaichen | Fixed in #11727 |
| `GlutenExplainSuite` | π’ | π’ | | Disabled in #12546 |
| `GlutenJoinHintSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenLogQuerySuite` | π’ | π’ | @kapilks | Fixed |
| `GlutenRandomDataGeneratorSuite` | π’ | π’ | #11919 | ixed in #12533 |
| `GlutenSetCommandSuite` | π’ | π’ | Fixed in #12533 | |
| `GlutenSingleLevelAggregateHashMapSuite` | π’ | π’ | @kapilks | Fixed |
| `GlutenSparkSessionJobTaggingAndCancellationSuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenTwoLevelAggregateHashMapSuite` | π’ | π’ | @kapilks | Fixed |
| `GlutenTwoLevelAggregateHashMapWithVectorizedMapSuite` | π’ | π’ | @kapilks | Fixed |
| `GlutenVariantEndToEndSuite` | π’ | π’ | @baibaichen | Fixed in #11726 |
| `GlutenVariantShreddingSuite` | π’ | π’ | @baibaichen | Fixed in #11726 |
| `GlutenXmlFunctionsSuite` | π’ | π’ | @baibaichen | Fixed in #11725 |
| `GlutenSparkSessionExtensionSuite` | π’ | π’ | | Fixed in #12475 |
| `GlutenTPCDSV1_4_PlanStabilitySuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCDSV1_4_PlanStabilityWithStatsSuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCDSV2_7_PlanStabilitySuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCDSV2_7_PlanStabilityWithStatsSuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCDSModifiedPlanStabilitySuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCDSModifiedPlanStabilityWithStatsSuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
| `GlutenTPCHPlanStabilitySuite` | π’ | π’ | @baibaichen | Fixed in #11799 |
### catalyst.expressions
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenCastWithAnsiOnSuite` | π΄ (4 failures) | π΄ (10 failures) | #10134 | ANSI + TimeType |
| `GlutenCollationRegexpExpressionsSuite` | π’ | π΄ (1 failure) | #11913 | Velox split limit unsupported |
| `GlutenCsvExpressionsSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenExpressionEvalHelperSuite` | π’ | π’ | | Disabled in #12484 |
| `GlutenObjectExpressionsSuite` | π’ | π’ | | Disabled in #12484 |
| `GlutenOrderingSuite` | π’ | π’ | #11919 | Fixed in #12484 |
| `GlutenScalaUDFSuite` | π’ | π’ | | Disabled in #12484 |
| `GlutenToPrettyStringSuite` | π’ | π’ | #11918 | Fixed in #12048 |
| `GlutenXmlExpressionsSuite` | π’ | π’ | @baibaichen | Fixed in #11580 |
### connector
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenGroupBasedUpdateTableSuite` | π’ | π’ | #11912 | Fixed in #12493 |
| `GlutenMergeIntoDataFrameSuite` | π’ | π’ | @baibaichen | Fixed in #11812 |
### execution
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenColumnarRulesSuite` | π’ | π’ | #11920 | Fixed in #12004 |
| `GlutenDataSourceScanExecRedactionSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenDataSourceV2ScanExecRedactionSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenExternalAppendOnlyUnsafeRowArraySuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenHiveResultSuite` | π’ | π’ | #11919 | Fixed in #12484 |
| `GlutenInsertSortForLimitAndOffsetSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenLogicalPlanTagInSparkPlanSuite` | π’ | π’ | @baibaichen | Fixed in #11833. Core fix: propagate LOGICAL_PLAN_TAG during offload |
| `GlutenMultiStatefulOperatorsSuite` | π’ | π’ | #11911 | Fixed in #11957 |
| `GlutenPlannerSuite` | π’ | π’ | | Disabled in #12512 |
| `GlutenProjectedOrderingAndPartitioningSuite` | π’ | π’ | | Disabled in #12512 |
| `GlutenRemoveRedundantProjectsSuite` | π’ | π’ | | Fixed in #12506 |
| `GlutenRemoveRedundantSortsSuite` | π’ | π’ | | Fixed in #12497 |
| `GlutenSQLExecutionSuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenSQLJsonProtocolSuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenShufflePartitionsUtilSuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenSimpleSQLViewSuite` | π’ | π’ | #11912 | Fixed in #12502 |
| `GlutenSparkPlanSuite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenUnsafeRowSerializerSuite` | π’ | π’ | @baibaichen | Fixed in #11847 |
| `GlutenWholeStageCodegenSparkSubmitSuite` | π’ | π’ | | Disabled in #12512 |
| `GlutenWholeStageCodegenSuite` | π’ | π’ | | Fixed in #12507 |
### execution.joins
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenSingleJoinSuite` | π’ | π’ | @baibaichen | Fixed in #11577 |
### execution.datasources.parquet
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenParquetTypeWideningSuite` | π’ | π’ | @baibaichen | Fixed in #11683 |
| `GlutenParquetVariantShreddingSuite` | π’ | π’ |@baibaichen | Fixed in #11726 |
### execution.datasources.text
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenWholeTextFileV1Suite` | π’ | π’ | @baibaichen | Fixed in #11816 |
| `GlutenWholeTextFileV2Suite` | π’ | π’ | @baibaichen | Fixed in #11816 |
### execution.python
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenPythonDataSourceSuite` | π΄ (1 failures) | π΄ | @baibaichen | #11908 |
| `GlutenPythonUDFSuite` | π΄ (1 failures) | π΄ | @baibaichen | #11908 |
| `GlutenPythonUDTFSuite` | π΄ | π΄ | @baibaichen | #11908 |
| `GlutenRowQueueSuite` | π’ | π΄ | @baibaichen | #11908 |
### sources
| Suite | Spark 4.0 | Spark 4.1 | Owner | Comments |
|-------|-----------|-----------|-------|----------|
| `GlutenBucketedReadWithHiveSupportSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenBucketedWriteWithHiveSupportSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenCommitFailureTestRelationSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenDisableUnnecessaryBucketedScanWithHiveSupportSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenJsonHadoopFsRelationSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenParquetHadoopFsRelationSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
| `GlutenSimpleTextHadoopFsRelationSuite` | π’ | π’ | @baibaichen | Deleted in #11847 (TestHiveSingleton conflict) |
### streaming
Structured Streaming suites (20 disabled) are tracked in #11911.
---
## Summary Statistics
- **Total suites disabled in Spark 4.0**: 69
- **Total suites disabled in Spark 4.1**: 79
- **Unique suites across both versions**: 79
### PRs
| PR | Status | Suites fixed |
|----|--------|-------------|
| #11577 | β
Merged | 1: SingleJoinSuite |
| #11580 | β
Merged | 1: XmlExpressionsSuite |
| #11683 | β
Merged | 1: ParquetTypeWideningSuite |
| #11725 | β
Merged | 1: XmlFunctionsSuite |
| #11726 | β
Merged | 2: VariantEndToEnd, VariantShreddingSuite |
| #11727 | β
Merged | 1: DataFrameSubquerySuite |
| #11799 | β
Merged | 7: TPCDS PlanStability suites |
| #11812 | β
Merged | 1: MergeIntoDataFrame (JobTagging reverted in #11908) |
| #11833 | β
Merged | 1: LogicalPlanTagInSparkPlan (150 tests) |
| #11847 | β
Merged | 6 enabled + 7 deleted: SystemPropertyTrait suites |
| #11816 | β
Merged | 8 suites + 2 trait fix + 1 disable |
| #11908 | π΅ Draft | 4: PythonDataSource, PythonUDF, PythonUDTF, RowQueue |
Contributor guide
Assessment
This issue has not been assessed yet.