aws / aws/sagemaker-python-sdk

`DeserializerWrapper` prevents the use of valid input Content-Types during inference - SageMaker endpoitns build with ModelBuilder and SchemaBuilder

Abierto
#5,006 0 comentarios 0 reacciones 1 asignado Reclamado por @rsareddy0329 Ver en GitHub
component: Inference APIs and Interfaces type: bug
Lenguaje dominante
Python
Estrellas
2.3k
Forks
1.3k
Merge medio
1 d 22 h
PR fusionados (30 d)
35

Descripción

**Describe the bug**
[`DeserializerWrapper`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/serve/builder/schema_builder.py#L81) overwrites the `content_type` provided to `deserialize()` to be the value of `Accept`. This `DeserializerWrapper` is used for both Input and Output de-serialization when using [`SchemaBuilder`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/serve/builder/schema_builder.py#L134). The comments mention:

> We need to overwrite the accept type because model servers like XGBOOST always returns "text/html"

but this effectively prevents endpoints deployed through the use of [`ModelBuilder`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/serve/builder/model_builder.py#L139) and `SchemaBuilder` to support additional `Content-Types`, despite the various implementations of [`BaseDeserializer`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/base_deserializers.py#L191) supporting multiple Content-Types.

For example, an inference endpoint deployed via `ModelBuilder` and `SchemaBuilder` which takes a `np.array` as input cannot be invoked with a `Content-Type` such as `application/json` or `text/csv` due to this overwriting. This also makes the developer experience confusing as the stack trace shows the execution path for `application/x-npy` Content-Type, despite a different Content-Type being explicitly provided.

**To reproduce**
```py
import sagemaker

schema_builder = sagemaker.serve.SchemaBuilder(
sample_input=np.array([[6.4, 2.8, 5.6, 2.2]]),
sample_output=np.array(
[[0.09394703, 0.4797692, 0.42628378]], dtype=np.float32
),
)
# similar usage to that defined in unit tests for base NumpyDeserializer: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/unit/sagemaker/deserializers/test_deserializers.py#L113
schema_builder.input_deserializer.deserialize(io.BytesIO(b"[[6.4, 2.8, 5.6, 2.2]]"), content_type="application/json")

# Below is not needed to reproduce, but is provided as an example for how I am using SchemaBuilder
model_builder = sagemaker.serve.ModelBuilder(
mode=sagemaker.serve.mode.function_pointers.Mode.SAGEMAKER_ENDPOINT,
model_metadata={
"MLFLOW_MODEL_PATH": f"models:/{model_name}/{model_version}",
"MLFLOW_TRACKING_ARN": TRACKING_SERVER_ARN,
},
model_server=sagemaker.serve.ModelServer.TENSORFLOW_SERVING,
role_arn=sagemaker.get_execution_role(),
schema_builder=schema_builder,
)
model = model_builder.build()
model.deploy(
initial_instance_count=1,
instance_type="ml.m5.xlarge",
)
```

**Expected behavior**
I would like the `content_type` to not be overwritten for input de-serialization, so that I can use `SchemaBuilder` for inference endpoints while providing a Content-Type other than `application/x-npy`, such as `application/json` via CURL.

**Screenshots or logs**
```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/dillodon/.venv/lib/python3.12/site-packages/sagemaker/base_deserializers.py:231 in │
│ deserialize │
│ │
│ 228 │ │ │ │ return np.array(json.load(codecs.getreader("utf-8")(stream)), dtype=self │
│ 229 │ │ │ if content_type == "application/x-npy": │
│ 230 │ │ │ │ try: │
│ ❱ 231 │ │ │ │ │ return np.load(io.BytesIO(stream.read()), allow_pickle=self.allow_pi │
│ 232 │ │ │ │ except ValueError as ve: │
│ 233 │ │ │ │ │ raise ValueError( │
│ 234 │ │ │ │ │ │ "Please set the param allow_pickle=True \ │
│ │
│ /Users/dillodon/.venv/lib/python3.12/site-packages/numpy/lib/npyio.py:462 in load │
│ │
│ 459 │ │ else: │
│ 460 │ │ │ # Try a pickle │
│ 461 │ │ │ if not allow_pickle: │
│ ❱ 462 │ │ │ │ raise ValueError("Cannot load file containing pickled data " │
│ 463 │ │ │ │ │ │ │ │ "when allow_pickle=False") │
│ 464 │ │ │ try: │
│ 465 │ │ │ │ return pickle.load(fid, **pickle_kwargs) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Cannot load file containing pickled data when allow_pickle=False

During handling of the above exception, another exception occurred:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in :1 │
│ │
│ /Users/dillodon/.venv/lib/python3.12/site-packages/sagemaker/serve/builder/schema_builder.py:77 │
│ in deserialize │
│ │
│ 74 │ │
│ 75 │ def deserialize(self, stream, content_type: str = None): │
│ 76 │ │ """Deserialize stream into object""" │
│ ❱ 77 │ │ return self._deserializer.deserialize( │
│ 78 │ │ │ stream, │
│ 79 │ │ │ # We need to overwrite the accept type because model │
│ 80 │ │ │ # servers like XGBOOST always returns "text/html" │
│ │
│ /Users/dillodon/.venv/lib/python3.12/site-packages/sagemaker/base_deserializers.py:233 in │
│ deserialize │
│ │
│ 230 │ │ │ │ try: │
│ 231 │ │ │ │ │ return np.load(io.BytesIO(stream.read()), allow_pickle=self.allow_pi │
│ 232 │ │ │ │ except ValueError as ve: │
│ ❱ 233 │ │ │ │ │ raise ValueError( │
│ 234 │ │ │ │ │ │ "Please set the param allow_pickle=True \ │
│ 235 │ │ │ │ │ │ to deserialize pickle objects in NumpyDeserializer" │
│ 236 │ │ │ │ │ ).with_traceback(ve.__traceback__) │
│ │
│ /Users/dillodon/.venv/lib/python3.12/site-packages/sagemaker/base_deserializers.py:231 in │
│ deserialize │
│ │
│ 228 │ │ │ │ return np.array(json.load(codecs.getreader("utf-8")(stream)), dtype=self │
│ 229 │ │ │ if content_type == "application/x-npy": │
│ 230 │ │ │ │ try: │
│ ❱ 231 │ │ │ │ │ return np.load(io.BytesIO(stream.read()), allow_pickle=self.allow_pi │
│ 232 │ │ │ │ except ValueError as ve: │
│ 233 │ │ │ │ │ raise ValueError( │
│ 234 │ │ │ │ │ │ "Please set the param allow_pickle=True \ │
│ │
│ /Users/dillodon/.venv/lib/python3.12/site-packages/numpy/lib/npyio.py:462 in load │
│ │
│ 459 │ │ else: │
│ 460 │ │ │ # Try a pickle │
│ 461 │ │ │ if not allow_pickle: │
│ ❱ 462 │ │ │ │ raise ValueError("Cannot load file containing pickled data " │
│ 463 │ │ │ │ │ │ │ │ "when allow_pickle=False") │
│ 464 │ │ │ try: │
│ 465 │ │ │ │ return pickle.load(fid, **pickle_kwargs) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Please set the param allow_pickle=True to deserialize pickle objects in NumpyDeserializer
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.237.3
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: TensorFlow (but applies to all)
- **Framework version**: 2.16 (applies to all)
- **Python version**: 3.12.7
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
`model.deploy()` returns an instance of [`Predictor`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/base_predictor.py#L98), which does work properly. However, this limits inference to Python clients who have created a `Predictor` instance. Clients in other languages are unable to invoke the inference server, for example.

Workarounds may be possible by implementing [`CustomPayloadTranslator`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/serve/marshalling/custom_payload_translator.py#L11) and providing it via [`input_translator`](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/serve/builder/schema_builder.py#L117C9-L117C25), but I have not yet tested this.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.