aws / aws/amazon-sagemaker-examples

insufficient Disk space error in p3.8xlarge.

Open
#3,478 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am facing insufficient diskspace error when I execute the code even though I have used Volume_size = 250

from sagemaker.tensorflow import TensorFlow

# volume_size=200,

model_path = f"s3://{sagemaker_session.default_bucket()}/Report"

tf_estimator = TensorFlow(
entry_point='yolo_object_detection_train_with_fixed.py',
role=role,
instance_count=1,
instance_type='ml.p3.8xlarge',
framework_version='2.3.1',
py_version='py37',
script_mode=True,
sagemaker_session=sagemaker_session,
output_path=model_path,
volume_size=250,
)

when i used shutil to check my container memory i am getting only 25GB but i have given volume_size=250,
import shutil

total, used, free = shutil.disk_usage("/")
print("Total: %d GiB" % (total // (2**30)))
print("Used: %d GiB" % (used // (2**30)))
print("Free: %d GiB" % (free // (2**30)))

Total: 25 GiB
Used: 5 GiB
Free: 18 GiB

Contributor guide

Open the contributing guide

Research direction

Start with the TensorFlow estimator configuration in the issue and the referenced yolo_object_detection_train_with_fixed.py entry point. Reproduce the disk-usage report in an ml.p3.8xlarge SageMaker training job, then inspect the SageMaker TensorFlow volume_size documentation and container mount behavior. Done requires identifying whether the example or configuration is wrong and documenting or correcting the supported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.