dotnet / dotnet/spark

ClassNotFoundException when using SparkBatchClient

Open
#1,071 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

My goal is to trigger a .NET Spark job programmatically. I'm following this [tutorial ](https://github.com/Azure/azure-sdk-for-net/blob/Azure.Analytics.Synapse.Artifacts_1.0.0-preview.15/sdk/synapse/Azure.Analytics.Synapse.Spark/samples/Sample1_SubmitSparkJobAsync.md), but I keep getting ClassNotFoundException (for my WordCount C# class).
It works only when I trigger the Spark job using Synapse Studio UI (by creating an Apache Spark Job definition).

The .zip file for the .NET application has the .dll for the class.

**Code**

```
SparkBatchClient client = new SparkBatchClient(new Uri(endpoint), sparkPoolName, new DefaultAzureCredential());

string name = "name";
string file = "abfss://xxx.core.windows.net/jobs/WordCount.zip";

SparkBatchJobOptions request = new SparkBatchJobOptions(name, file)
{
ClassName = "WordCount",
DriverMemory = "28g",
DriverCores = 4,
ExecutorMemory = "28g",
ExecutorCores = 4,
ExecutorCount = 2
};

SparkBatchOperation createOperation = await client.StartCreateSparkBatchJobAsync(request);
SparkBatchJob jobCreated = await createOperation.WaitForCompletionAsync();
```
---------------------------------------

**Error**
Livy:
`22/06/14 20:36:58 WARN DependencyUtils: Skip remote jar abfss://xxx.core.windows.net/jobs/WordCount.zip.`

Driver (stderr):
```
22/06/13 19:54:27 ERROR ApplicationMaster: Uncaught exception:
java.lang.ClassNotFoundException: WordCount
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:722)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:496)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:905)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:904)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:904)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.