aws / aws/sagemaker-huggingface-inference-toolkit

How to access CustomAttributes in async inferece request input_fn

Open
#89 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

I need to access [CustomAttributes](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpointAsync.html) in async request.

I tried to access it inside a custom `input_fn`

`def input_fn(self, input_data, content_type):`

self.context coud give access to custom [properties](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/main/src/sagemaker_huggingface_inference_toolkit/handler_service.py#LL203C28-L203C35)

but I am not sure about this
```
try:
if not self.initialized:
self.initialize(context)
```
(I am new to python). But this looks like it would instantiate the class once and fill the context once, so it would not get update on each request with the additional context params i guess?!

anyhow. it looks like `self`
will not be accessible inside
`def input_fn(self, input_data, content_type):`
hence no access to additional params.
@philschmid Any idea about how to access additional CustomAttributes ?

Contributor guide

Open the contributing guide

Research direction

Read src/sagemaker_huggingface_inference_toolkit/handler_service.py around context initialization, then trace how the async request reaches the custom input_fn(input_data, content_type) entry point. Determine whether CustomAttributes are refreshed per request; done means the issue has a documented and tested access path, or a clear explanation that the current hook cannot expose them.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.