apache / apache/gluten

[CORE] Refactor in Spark to make shims and test easier integration

Open
#11,216 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Like this PR [[SPARK-54301][SQL][TESTS] Enhance Spark SQL test suites for easier integration with other projects #52996](https://github.com/apache/spark/pull/52996), Spark community is open to accept such refactor to make external project easy to integrate, then we don't need to copy and rewrite so many tests.

GlutenDateExpressionsSuite and other tests copy the test from `Replaced by a gluten test to pass timezone through config`.

```
withSQLConf(
SQLConf.SESSION_LOCAL_TIMEZONE.key -> zid.getId
) {
```

ParquetThriftCompatibilitySuite make field parquetFilePath protected or read from spark.test.home or $SPARK_HOME
StreamingQuerySuite setUp2dot4dot0Checkpoint read file
FileBasedDataSourceSuite test `Option recursiveFileLookup: disable partition inferring` dataPath

FileSourceScanExecShim and AbstractBatchScanExec copies most from FileSourceScanExec and BatchScanExec, if we can make them abstract, we don't need to copy them, and inputRDD[ROW] is not used in Gluten, we do a refactor to extract the finalPartitions that construct inputRDD, if Spark supply a function to getFinalPartitions and call it in inputRDD, we don't need to copy them, and the copy may cause unit test failure when upgrade Spark version

GlutenReplaceHashWithSortAggSuite make function checkNumAggs protected

BucketedReadSuite make testBucketing, getFileScan protected

DisableUnnecessaryBucketedScanSuite make checkDisableBucketedScan or checkNumBucketedScan protected

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Spark SQL test-suite refactor and the named Gluten suites and shims, including FileSourceScanExecShim and AbstractBatchScanExec. Trace which copied methods and test data paths are required for external integration. Done means the listed tests and shims no longer need the described duplicated or inaccessible pieces while preserving their test coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
distributed-systems, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.