aws / aws/sagemaker-python-sdk

need to be able to set return data size for each request in batch transform

Cerrado
#1,882 13 comentarios 1 reacción 1 asignado Reclamado por @mollyheamazon Ver en GitHub
component: Inference APIs and Interfaces type: feature request
Lenguaje dominante
Python
Estrellas
2.3k
Forks
1.3k
Merge medio
1 d 22 h
PR fusionados (30 d)
35

Descripción

**Describe the feature you'd like**
I'm running a NLP inference job to get sentence embedding vector for each record. Each record is less than 512 words only and the returned vector has 768 floats. Even if I set max_payload to 1 and max_concurrent_transforms to 1, I still got:
**io.netty.handler.codec.CorruptedFrameException: Message size exceed limit: 113038147**

I know there is a 5MB limit for endpoint inference. I'm relatively sure that this is caused by the size limit of return results of each request from batch transform, though I'm not able to find it in the doc.

I hope you can
1. Clarify the size limit for each request for batch transform in the doc.
2. Mimicking max_payload, expose a parameter to let user control the return data size for each request for batch transform.

**How would this feature be used? Please describe.**

Add a max_return_payload parameter to model.transformer like below.

tfm = model.transformer(instance_count=1,
instance_type='ml.p3.2xlarge',
accept='text/json',
assemble_with='Line',
output_path=batch_out,
strategy='MultiRecord',
max_payload=3,
**max_return_payload=30**)

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Use strategy='SingleRecord' can bypass the issue. But it is significant slower as it's not making use of parallelism. Setting max_concurrent_transform to a larger value can make use of parallelism. But it could cause problem for code that's not designed for concurrency.

**Additional context**
Add any other context or screenshots about the feature request here.

A similar issue is raised here by other people: https://github.com/aws/sagemaker-python-sdk/issues/1096

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.