ClassNotFoundException when using SparkBatchClient
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
My goal is to trigger a .NET Spark job programmatically. I'm following this [tutorial ](https://github.com/Azure/azure-sdk-for-net/blob/Azure.Analytics.Synapse.Artifacts_1.0.0-preview.15/sdk/synapse/Azure.Analytics.Synapse.Spark/samples/Sample1_SubmitSparkJobAsync.md), but I keep getting ClassNotFoundException (for my WordCount C# class).
It works only when I trigger the Spark job using Synapse Studio UI (by creating an Apache Spark Job definition).
The .zip file for the .NET application has the .dll for the class.
**Code**
```
SparkBatchClient client = new SparkBatchClient(new Uri(endpoint), sparkPoolName, new DefaultAzureCredential());
string name = "name";
string file = "abfss://xxx.core.windows.net/jobs/WordCount.zip";
SparkBatchJobOptions request = new SparkBatchJobOptions(name, file)
{
ClassName = "WordCount",
DriverMemory = "28g",
DriverCores = 4,
ExecutorMemory = "28g",
ExecutorCores = 4,
ExecutorCount = 2
};
SparkBatchOperation createOperation = await client.StartCreateSparkBatchJobAsync(request);
SparkBatchJob jobCreated = await createOperation.WaitForCompletionAsync();
```
---------------------------------------
**Error**
Livy:
`22/06/14 20:36:58 WARN DependencyUtils: Skip remote jar abfss://xxx.core.windows.net/jobs/WordCount.zip.`
Driver (stderr):
```
22/06/13 19:54:27 ERROR ApplicationMaster: Uncaught exception:
java.lang.ClassNotFoundException: WordCount
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:722)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:496)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:905)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:904)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:904)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
```
Contributor guide
Assessment
This issue has not been assessed yet.