aws / aws/sagemaker-huggingface-inference-toolkit

Better Documentation for Custom Inference and HF_MODEL_ID

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

It's not at all clear if the `HuggingFaceModel` class from this library honors the `entry_point ` and `source_dir` when image_uri is specified. I have tried this with several containers and the only way I can run my custom `code/inference.py` is by archiving the model and providing the s3 path in ` model_data=s3_model_uri`. Providing the `HF_MODEL_ID` and `inference.py` should work but from the logs I can see that's not the case because there is nothing about loading the inference.py or when I make a call to the model my logs are not there but if I archive this model and upload to s3 it works.

This is not documented. It's also not intuitive.

Contributor guide

Open the contributing guide

Research direction

Start with the HuggingFaceModel behavior around entry_point, source_dir, image_uri, HF_MODEL_ID, and model_data, using the reported code/inference.py and container logs as the reproduction path. Document whether custom inference is honored when image_uri and HF_MODEL_ID are used, and explain the supported setup and expected logging behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.