aws / aws/amazon-sagemaker-examples

PySparkProcessor - Unable to locate credentials for boto3 call in AppMaster

Open
#1,689 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am curious if there are issues calling boto3 client from AppMaster when the network is isolated. Or, maybe my network config is off... not sure. I am running pyspark script using the PySparkProcessor (injected into the interface). That script needs to pull objects from s3 as it runs in the AppMaster\Client before working with spark session across the slave workers. I have network isolation turned on, and security groups set.. I have allowed traffic to s3.

When I turn network isolation I get the following.
"raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials"

This traces back to this line of code in my pyspark script I inject.
s3 = boto3.client('s3')

I am using the same role my sagemaker notebook is running in.. that allows me to make these calls to boto3. I simply pass that role to.
networkConfig = NetworkConfig(enable_network_isolation=True,
security_group_ids=[sg_s3_access,
sg_master,
sg_slaves],
subnets=[private_subnet_3])
role = sagemaker.get_execution_role()

spark_processor = PySparkProcessor(base_job_name="some-job",
role=role,
instance_count=2,
instance_type="ml.m5.4xlarge",
max_runtime_in_seconds=2400,
network_config=networkConfig,
image_uri="............dkr.ecr.us-east-1.amazonaws.com/sagemaker-spark-processing:2.4-cpu-py37-v1.0")

All works ok when "enable_network_isolation=False" and I still pass in my networkconfig. Is there a defect in calling boto3 from PySpark script from AppMaster when network isolation is turned on? Or, should I look at my network config again?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.