kubeflow / kubeflow/spark-operator

Unable to run a job with main file point to s3 bucket

Open
#2,301 5 comments 8 reactions 0 assignees View on GitHub
lifecycle/frozen
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What question do you want to ask?

- [ Y ] ✋ I have searched the open/closed issues and my issue is not listed.
Similar issue found: https://github.com/kubeflow/spark-operator/issues/996

Error
```
│ Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found │
│ at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688) │
│ at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431) │
│ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) │
│ at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) │
│ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) │
│ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) │
│ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) │
│ at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831) │
│ at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:727) │
│ at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264) │
│ at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:103) │
│ ... 18 more │
│ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found │
│ at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592) │
│ at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686) │
```

### Additional context
This is happening because spark-operator image doesn't have hadoop-aws jar. Is there a recommended way to pull jars from S3?

_No response_

### Have the same question?

Give it a 👍 We prioritize the question with most 👍

Contributor guide

Open the contributing guide

Research direction

Review the Spark Kubernetes dependency-download path named in the stack trace and inspect whether the operator image includes the hadoop-aws dependency. Confirm the supported way to make that dependency available for an S3 main file, then verify a job can start without the ClassNotFoundException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java, kubernetes, spark
Domain
cloud, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.