[SUPPORT] Remote connection issue while testing locally Apache Hudi with Glue Image and LocalStack
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
I am trying to run some tests in Docker using the image `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and `localstack` for AWS environment. No matter what, every time the tests run, Hudi tries to connect to the remote AWS instead of pointing to LocalStack. This is the configuration I am using at the moment:
```
packages = [
'/home/glue_user/spark/jars/spark-avro_2.12-3.3.0-amzn-1.jar',
'/home/glue_user/aws-glue-libs/datalake-connectors/hudi-0.12.1/hudi-spark3-bundle_2.12-0.12.1.jar',
'/home/glue_user/aws-glue-libs/jars/aws-java-sdk-1.12.128.jar',
'/home/glue_user/aws-glue-libs/jars/aws-java-sdk-glue-1.12.128.jar',
'/home/glue_user/spark/jars/hadoop-aws-3.3.3-amzn-0.jar',
]
conf = SparkConf() \
.set('spark.jars', ','.join(packages))\
.set('spark.serializer', 'org.apache.spark.serializer.KryoSerializer')\
.set('spark.sql.catalog.spark_catalog', 'org.apache.spark.sql.hudi.catalog.HoodieCatalog')\
.set('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension')
spark_context = SparkContext(conf=conf)
glue_context = GlueContext(spark_context)
spark_session = glue_context.spark_session
# HUDI S3 ACCESS
spark_session.conf.set('fs.defaultFS', 's3://mybucket')
spark_session.conf.set('fs.s3.awsAccessKeyId', 'test')
spark_session.conf.set('fs.s3.awsSecretAccessKey', 'test')
spark_session.conf.set('fs.s3a.awsAccessKeyId', 'test')
spark_session.conf.set('fs.s3a.awsSecretAccessKey', 'test')
spark_session.conf.set('fs.s3a.endpoint', 'http://localstack:4566')
spark_session.conf.set('fs.s3a.connection.ssl.enabled', 'false')
spark_session.conf.set('fs.s3a.path.style.access', 'true')
spark_session.conf.set('fs.s3a.signing-algorithm', 'S3SignerType')
spark_session.conf.set('spark.sql.legacy.setCommandRejectsSparkCoreConfs', 'false')
# SPARK CONF
spark_session.conf.set('spark.sql.shuffle.partitions', '2')
spark_session.conf.set('spark.sql.crossJoin.enabled', 'true')
```
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
5.
6.
**Expected behavior**
How can I make it to point to my local environment (http://localstack:4566) instead of AWS remote?
**Environment Description**
* Hudi version : 0.12
* Spark version : 3.3.0
* Hive version :
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : yes
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
```
An error occurred while calling o1719.save.
: java.nio.file.AccessDeniedException: s3://mybucket/myzone/location/.hoodie: getFileStatus on s3://mybucket/myzone/location/.hoodie: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: B1ZJ8JDPY2HX514F; S3 Extended Request ID: rzBqoLQxJb4PSKNW+uCbyVCbqYtpCB0aFHvX7JWTCDJ/PTfQdgESAkOzxWR6aPua8OhuEcajIM8=; Proxy: null), S3 Extended Request ID: rzBqoLQxJb4PSKNW+uCbyVCbqYtpCB0aFHvX7JUTVIJ/PTfQdgEHNkOzxWR6aPua8OhuEcajIM8=:403 Forbidden
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin with the Docker setup and the SparkConf configuration shown in the report, then reproduce the save operation against LocalStack at http://localstack:4566. Trace the S3 settings used by the failing .hoodie getFileStatus call and compare them with the configured endpoint; done means the test uses LocalStack rather than reaching remote AWS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, java, spark
- Domain
- cloud, data-engineering, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100