kubeflow / kubeflow/spark-operator
Unable to run a job with main file point to s3 bucket
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
### What question do you want to ask?
- [ Y ] ✋ I have searched the open/closed issues and my issue is not listed.
Similar issue found: https://github.com/kubeflow/spark-operator/issues/996
Error
```
│ Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found │
│ at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688) │
│ at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431) │
│ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) │
│ at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) │
│ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) │
│ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) │
│ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) │
│ at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831) │
│ at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:727) │
│ at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264) │
│ at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:103) │
│ ... 18 more │
│ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found │
│ at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592) │
│ at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686) │
```
### Additional context
This is happening because spark-operator image doesn't have hadoop-aws jar. Is there a recommended way to pull jars from S3?
_No response_
### Have the same question?
Give it a 👍 We prioritize the question with most 👍
Contributor guide
Research direction
Review the Spark Kubernetes dependency-download path named in the stack trace and inspect whether the operator image includes the hadoop-aws dependency. Confirm the supported way to make that dependency available for an S3 main file, then verify a job can start without the ClassNotFoundException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java, kubernetes, spark
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100