while trying to read .csv to a spark dataframe using conda_python3 kernel in sagemaker AWS. getting an error
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Please fill out the form below.
### System Information
- **Spark or PySpark**:
- **SDK Version**:
- **Spark Version**:
- **Algorithm (e.g. KMeans)**:
### Describe the problem
while trying to read .csv to a spark dataframe using conda_python3 kernel in sagemaker AWS. getting an error. How t read a file from s3. I have tried using boto3, still getting the same error. If any changes has to be done using shell please specify the steps to do it in AWS an I am new to AWS.
### Minimal repo / logs
Py4JavaError: An error occured while calling o138.csv
: java.io.IOException: No FileSystem for scheme: s3
.
.
.
- **Exact command to reproduce**:
Contributor guide
Research direction
No repository file, test, version, or exact command is provided. Start by reproducing the CSV read in the conda_python3 kernel on SageMaker with the reported S3 path, capturing the complete Py4JavaError and Spark configuration; done means a minimal reproducible case and a confirmed way to read the S3 CSV without the “No FileSystem for scheme: s3” error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100