aws / aws/sagemaker-huggingface-inference-toolkit

Make DEFAULT_HF_HUB_MODEL_EXPORT_DIRECTORY configurable

Open
#115 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

In [```DEFAULT_HF_HUB_MODEL_EXPORT_DIRECTORY = os.path.join(os.getcwd(), ".sagemaker/mms/models")```](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/80634b30703e8e9525db8b7128b05f713f42f9dc/src/sagemaker_huggingface_inference_toolkit/mms_model_server.py#L48) the directory is forced to be in the same path as the current directory of the running process. In some SageMaker instances this is a relatively small partition that can't be extended. Allowing this var to be modified by an environment variable will allow the download of larger models in a variety of instances (i.e. ml.g5.16xlarge)

To reproduce the problem you can try this particular model (other large models will fail the same):

```
hub = {
'HF_MODEL_ID':'Salesforce/instructblip-flan-t5-xxl',
'HF_TASK':'image-to-text',
'SM_NUM_GPUS': '1',
'HF_HOME':'/tmp/hf_home',
'HF_ASSETS_CACHE': '/tmp/hf_assets_cache',
'HF_DATASETS_CACHE':'/tmp/hf_cache',
'HF_DATASETS_HOME':'/tmp/hf_home',
'HF_HUB_CACHE': '/tmp/hf_hub_cache'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
transformers_version='4.37.0',
pytorch_version='2.1.0',
py_version='py310',
env=hub,
role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type='ml.g5.16xlarge', # ec2 instance type
# volume_size=256
)
```

The error in CloudWatch is similar to:

```
OSError: [Errno 28] No space left on device: '/tmp/hf_hub_cache/tmpd1hcphh0' -> '/.sagemaker/mms/models/Salesforce__instructblip-flan-t5-xxl/pytorch_model-00001-of-00005.bin'
```

Contributor guide

Open the contributing guide

Research direction

Start in src/sagemaker_huggingface_inference_toolkit/mms_model_server.py at DEFAULT_HF_HUB_MODEL_EXPORT_DIRECTORY and trace where that path is used during model download. Reproduce with the Salesforce/instructblip-flan-t5-xxl deployment configuration and verify that an environment-provided directory prevents the no-space-left-on-device failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.