aws / aws/sagemaker-spark

Wrong version of aws-java-sdk-bundle in sagemaker-spark 1.4.5

Open
#149 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

### System Information
- **Spark or PySpark**: 3.3.0
- **SDK Version**: 1.4.5
- **Spark Version**: 3.3.0

### Describe the problem
I just spend 3 days trying to fix this but to no avail. My setup on an AWS notebook instance:
jars:
aws-java-sdk-bundle-1.11.901.jar
aws-java-sdk-core-1.12.262.jar
aws-java-sdk-kms-1.12.262.jar
aws-java-sdk-s3-1.12.262.jar
aws-java-sdk-sagemaker-1.12.262.jar
aws-java-sdk-sagemakerruntime-1.12.262.jar
aws-java-sdk-sts-1.12.262.jar
hadoop-aws-3.3.1.jar
sagemaker-spark_2.12-spark_3.3.0-1.4.5.jar

Problem:

- Upon reading a file from S3 this error is thrown
this is caused by a bug in the httpclient jar dependency of pyspark and is reported here: https://issues.apache.org/jira/browse/HADOOP-18159?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17554677#comment-17554677

Based on suggested workarounds in the article above I tried 4 things

1. upgrade `aws-java-sdk-bundle` to version 1.12.262 like the other jars → didn’t work
1. downgrade `httpclient` to version 4.5.10 → didn’t work
2. tried to set the `aws-java-sdk` to disable SSL certificate checking (https://github.com/aws/aws-sdk-java-v2/issues/1786 ) → didn’t work with "-Dcom.amazonaws.sdk.disableCertChecking=true"
3. try to read from a bucket that doesn’t contain dots (.) → works

### Minimal repo / logs
`22/08/30 11:00:22 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: s3a://comp.data.sci.data.tst/some/folder/export_date=20220822.
org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://comp.data.sci.data.tst/some/folder/export_date=20220822: com.amazonaws.SdkClientException: Unable to execute HTTP request: Certificate for doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]: Unable to execute HTTP request: Certificate for doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:208)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3351)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
at org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4277)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:54)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
at scala.Option.getOrElse(Option.scala:189) `

- **Exact command to reproduce**:
Works:
`df = spark.read.parquet("s3a://aws-bucket-with-dashes/file_0_1_0.snappy.parquet")`
Doesn't work:
`df = spark.read.parquet("s3a://aws.bucket.with.dots/file_0_1_0.snappy.parquet")`

It's not possible to rename the bucket due to the many data consumers that depend on them.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two exact Spark parquet reads with the listed Spark, Hadoop, and AWS SDK JARs, then inspect the dependency versions involved in the S3A certificate error. Done means reading a bucket with dots succeeds without the certificate mismatch and the required aws-java-sdk-bundle version is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala, spark
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.