apache / apache/hudi

[SUPPORT] Remote connection issue while testing locally Apache Hudi with Glue Image and LocalStack

Open
#8,691 6 comments 1 reaction 0 assignees View on GitHub
area:aws engine:spark
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

I am trying to run some tests in Docker using the image `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and `localstack` for AWS environment. No matter what, every time the tests run, Hudi tries to connect to the remote AWS instead of pointing to LocalStack. This is the configuration I am using at the moment:

```
packages = [
'/home/glue_user/spark/jars/spark-avro_2.12-3.3.0-amzn-1.jar',
'/home/glue_user/aws-glue-libs/datalake-connectors/hudi-0.12.1/hudi-spark3-bundle_2.12-0.12.1.jar',
'/home/glue_user/aws-glue-libs/jars/aws-java-sdk-1.12.128.jar',
'/home/glue_user/aws-glue-libs/jars/aws-java-sdk-glue-1.12.128.jar',
'/home/glue_user/spark/jars/hadoop-aws-3.3.3-amzn-0.jar',
]

conf = SparkConf() \
.set('spark.jars', ','.join(packages))\
.set('spark.serializer', 'org.apache.spark.serializer.KryoSerializer')\
.set('spark.sql.catalog.spark_catalog', 'org.apache.spark.sql.hudi.catalog.HoodieCatalog')\
.set('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension')

spark_context = SparkContext(conf=conf)
glue_context = GlueContext(spark_context)
spark_session = glue_context.spark_session

# HUDI S3 ACCESS
spark_session.conf.set('fs.defaultFS', 's3://mybucket')
spark_session.conf.set('fs.s3.awsAccessKeyId', 'test')
spark_session.conf.set('fs.s3.awsSecretAccessKey', 'test')
spark_session.conf.set('fs.s3a.awsAccessKeyId', 'test')
spark_session.conf.set('fs.s3a.awsSecretAccessKey', 'test')
spark_session.conf.set('fs.s3a.endpoint', 'http://localstack:4566')
spark_session.conf.set('fs.s3a.connection.ssl.enabled', 'false')
spark_session.conf.set('fs.s3a.path.style.access', 'true')
spark_session.conf.set('fs.s3a.signing-algorithm', 'S3SignerType')
spark_session.conf.set('spark.sql.legacy.setCommandRejectsSparkCoreConfs', 'false')

# SPARK CONF
spark_session.conf.set('spark.sql.shuffle.partitions', '2')
spark_session.conf.set('spark.sql.crossJoin.enabled', 'true')
```

**To Reproduce**

Steps to reproduce the behavior:

1.
2.
3.
4.
5.
6.

**Expected behavior**

How can I make it to point to my local environment (http://localstack:4566) instead of AWS remote?

**Environment Description**

* Hudi version : 0.12

* Spark version : 3.3.0

* Hive version :

* Hadoop version : 3.3.3

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : yes

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

```
An error occurred while calling o1719.save.
: java.nio.file.AccessDeniedException: s3://mybucket/myzone/location/.hoodie: getFileStatus on s3://mybucket/myzone/location/.hoodie: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: B1ZJ8JDPY2HX514F; S3 Extended Request ID: rzBqoLQxJb4PSKNW+uCbyVCbqYtpCB0aFHvX7JWTCDJ/PTfQdgESAkOzxWR6aPua8OhuEcajIM8=; Proxy: null), S3 Extended Request ID: rzBqoLQxJb4PSKNW+uCbyVCbqYtpCB0aFHvX7JUTVIJ/PTfQdgEHNkOzxWR6aPua8OhuEcajIM8=:403 Forbidden
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with the Docker setup and the SparkConf configuration shown in the report, then reproduce the save operation against LocalStack at http://localstack:4566. Trace the S3 settings used by the failing .hoodie getFileStatus call and compare them with the configured endpoint; done means the test uses LocalStack rather than reaching remote AWS.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, java, spark
Domain
cloud, data-engineering, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.