[CORE] Refactor in Spark to make shims and test easier integration
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Like this PR [[SPARK-54301][SQL][TESTS] Enhance Spark SQL test suites for easier integration with other projects #52996](https://github.com/apache/spark/pull/52996), Spark community is open to accept such refactor to make external project easy to integrate, then we don't need to copy and rewrite so many tests.
GlutenDateExpressionsSuite and other tests copy the test from `Replaced by a gluten test to pass timezone through config`.
```
withSQLConf(
SQLConf.SESSION_LOCAL_TIMEZONE.key -> zid.getId
) {
```
ParquetThriftCompatibilitySuite make field parquetFilePath protected or read from spark.test.home or $SPARK_HOME
StreamingQuerySuite setUp2dot4dot0Checkpoint read file
FileBasedDataSourceSuite test `Option recursiveFileLookup: disable partition inferring` dataPath
FileSourceScanExecShim and AbstractBatchScanExec copies most from FileSourceScanExec and BatchScanExec, if we can make them abstract, we don't need to copy them, and inputRDD[ROW] is not used in Gluten, we do a refactor to extract the finalPartitions that construct inputRDD, if Spark supply a function to getFinalPartitions and call it in inputRDD, we don't need to copy them, and the copy may cause unit test failure when upgrade Spark version
GlutenReplaceHashWithSortAggSuite make function checkNumAggs protected
BucketedReadSuite make testBucketing, getFileScan protected
DisableUnnecessaryBucketedScanSuite make checkDisableBucketedScan or checkNumBucketedScan protected
### Gluten version
None
Contributor guide
Research direction
Start by reading the linked Spark SQL test-suite refactor and the named Gluten suites and shims, including FileSourceScanExecShim and AbstractBatchScanExec. Trace which copied methods and test data paths are required for external integration. Done means the listed tests and shims no longer need the described duplicated or inaccessible pieces while preserving their test coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- distributed-systems, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100