aws / aws/sagemaker-pytorch-inference-toolkit

How do I access Custom Attributes from the model during inference?

Open
#111 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**What did you find confusing? Please describe.**
It is clear how to [invoke an endpoint with Custom Attributes ](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html#SageMakerRuntime.Client.invoke_endpoint) but it is completely unclear to me how one would access this information from the Pytorch SDK inference code. I can see for the Tensorflow SDK they have documentation for it and imply the [second argument of the input handler function will provide access to it](https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/README.md#prepost-processing) but it seems in the Pytorch framework it is just a string indicating the input content MIME type.

It would seem from the [source ](https://github.com/aws/sagemaker-inference-toolkit/blob/7fcb80561c0d87cf035e194a86f9fa5c6c9d9b33/src/sagemaker_inference/transformer.py#L126)that the `context` object is not passed to the transformation functions, so not accessible at all to the user.

**Describe how documentation can be improved**
Explanation of how to access provided custom attributes using the Pytorch Inference Toolkit

**Additional context**
I want to achieve what is being done here https://aws.amazon.com/marketplace/pp/prodview-5jlvp43tsn3ny?sr=0-1&ref_=beagle&applicationId=AWSMPContessa - namely the ability to provide a confidence threshold to an inference request. I would like to use a binary format for the body of the payload (image/jpeg) so I cannot provide parameters as part of the body itself like if the body were a JSON.

Contributor guide

Open the contributing guide

Research direction

Read src/sagemaker_inference/transformer.py at the linked transformation flow, then compare it with the linked TensorFlow serving documentation and the invoke_endpoint entry point. Document whether PyTorch input handlers can access Custom Attributes, including the supported way to provide them or the current limitation, and make the expected inference behavior unambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.