dotnet / dotnet/spark

Fix E2E tests for Spark 3.0.

Open
#657 2 comments 0 reactions 1 assignee Claimed by @imback82 View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

#348 enabled building/testing against Spark 3.0. The following tests have been disabled to track separately:

#### Microsoft.Spark.E2ETest.IpcTests
* [x] `BroadcastTests.*`
* [x] `DataFrameTests.TestDataFrameGroupedMapUdf` / `DataFrameTests.TestGroupedMapUdf`
* [x] `DataFrameTests.TestVectorUdf` / `DataFrameTests.TestDataFrameVectorUdf`

(#653) should address some of the above.

#### Microsoft.Spark.Extensions.Delta.E2ETest
* [x] `DeltaTableTests.TestSignatures`
* [x] `DeltaTableTests.TestTutorialScenario`

The following needs to be configured for `SparkSession`:
```
.Option("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.Option("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
```

#### Microsoft.Spark.Extensions.Hyperspace.E2ETest
* [ ] `HyperspaceTests.*`

This depends on [microsoft/hyperspace](https://github.com/microsoft/hyperspace) to support Spark 3.0 (tracked in https://github.com/microsoft/hyperspace/pull/85)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.