aws / aws/sagemaker-pytorch-inference-toolkit

Documentation for inference.py `transform_fn`

Open
#142 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**What did you find confusing? Please describe.**
Huggingface have [documented](https://huggingface.co/docs/sagemaker/inference) how to use the sagemaker pytorch inference API in order to host their models. They make it quite clear that you must supply `model_fn` and then either `transform_fn` or (`input_fn`, `predict_fn` and `output_fn`). By using `transform_fn` you can have fine control of batch size for example, allowing you to handle large requests (in particular I have an issue where my batch transform jobs continuously die because the minimum payload of 1MB is way to large for my model - due to the large intermediate matrices I..e probabilities = batch_szie x num_labels)

I cannot find any mention of `transform_fn` in the documentation - https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html

It is mentioned in passing in one of the examples - https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html

**Describe how documentation can be improved**
Document the use of `transform_fn` as an alternative to `input_fn`, `predict_fn` and `output_fn`

**Additional context**
[Add any other context or screenshots about the documentation request here.]

This is how I was aware of `transform_fn`:

https://aws.amazon.com/blogs/machine-learning/run-computer-vision-inference-on-large-videos-with-amazon-sagemaker-asynchronous-endpoints/

The I found this:

(https://huggingface.co/docs/sagemaker/inference)

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker PyTorch documentation at using_pytorch.html and compare its inference API coverage with the linked Hugging Face inference documentation and MNIST example. Document transform_fn as an alternative to input_fn, predict_fn, and output_fn, and consider the existing inference.py context; done means users can find and understand how to use transform_fn.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.