Fix E2E tests for Spark 3.0.
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
#348 enabled building/testing against Spark 3.0. The following tests have been disabled to track separately:
#### Microsoft.Spark.E2ETest.IpcTests
* [x] `BroadcastTests.*`
* [x] `DataFrameTests.TestDataFrameGroupedMapUdf` / `DataFrameTests.TestGroupedMapUdf`
* [x] `DataFrameTests.TestVectorUdf` / `DataFrameTests.TestDataFrameVectorUdf`
(#653) should address some of the above.
#### Microsoft.Spark.Extensions.Delta.E2ETest
* [x] `DeltaTableTests.TestSignatures`
* [x] `DeltaTableTests.TestTutorialScenario`
The following needs to be configured for `SparkSession`:
```
.Option("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.Option("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
```
#### Microsoft.Spark.Extensions.Hyperspace.E2ETest
* [ ] `HyperspaceTests.*`
This depends on [microsoft/hyperspace](https://github.com/microsoft/hyperspace) to support Spark 3.0 (tracked in https://github.com/microsoft/hyperspace/pull/85)
Contributor guide
Assessment
This issue has not been assessed yet.