aws / aws/amazon-sagemaker-examples
insufficient Disk space error in p3.8xlarge.
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I am facing insufficient diskspace error when I execute the code even though I have used Volume_size = 250
from sagemaker.tensorflow import TensorFlow
# volume_size=200,
model_path = f"s3://{sagemaker_session.default_bucket()}/Report"
tf_estimator = TensorFlow(
entry_point='yolo_object_detection_train_with_fixed.py',
role=role,
instance_count=1,
instance_type='ml.p3.8xlarge',
framework_version='2.3.1',
py_version='py37',
script_mode=True,
sagemaker_session=sagemaker_session,
output_path=model_path,
volume_size=250,
)
when i used shutil to check my container memory i am getting only 25GB but i have given volume_size=250,
import shutil
total, used, free = shutil.disk_usage("/")
print("Total: %d GiB" % (total // (2**30)))
print("Used: %d GiB" % (used // (2**30)))
print("Free: %d GiB" % (free // (2**30)))
Total: 25 GiB
Used: 5 GiB
Free: 18 GiB
Contributor guide
Research direction
Start with the TensorFlow estimator configuration in the issue and the referenced yolo_object_detection_train_with_fixed.py entry point. Reproduce the disk-usage report in an ml.p3.8xlarge SageMaker training job, then inspect the SageMaker TensorFlow volume_size documentation and container mount behavior. Done requires identifying whether the example or configuration is wrong and documenting or correcting the supported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, tensorflow
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100