aws / aws/sagemaker-python-sdk

need to be able to set return data size for each request in batch transform

已关闭
#1,882 13 条评论 1 个 reaction 已指派 1 人 已被 @mollyheamazon 认领 在 GitHub 查看
component: Inference APIs and Interfaces type: feature request
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**Describe the feature you'd like**
I'm running a NLP inference job to get sentence embedding vector for each record. Each record is less than 512 words only and the returned vector has 768 floats. Even if I set max_payload to 1 and max_concurrent_transforms to 1, I still got:
**io.netty.handler.codec.CorruptedFrameException: Message size exceed limit: 113038147**

I know there is a 5MB limit for endpoint inference. I'm relatively sure that this is caused by the size limit of return results of each request from batch transform, though I'm not able to find it in the doc.

I hope you can
1. Clarify the size limit for each request for batch transform in the doc.
2. Mimicking max_payload, expose a parameter to let user control the return data size for each request for batch transform.

**How would this feature be used? Please describe.**

Add a max_return_payload parameter to model.transformer like below.

tfm = model.transformer(instance_count=1,
instance_type='ml.p3.2xlarge',
accept='text/json',
assemble_with='Line',
output_path=batch_out,
strategy='MultiRecord',
max_payload=3,
**max_return_payload=30**)

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Use strategy='SingleRecord' can bypass the issue. But it is significant slower as it's not making use of parallelism. Setting max_concurrent_transform to a larger value can make use of parallelism. But it could cause problem for code that's not designed for concurrency.

**Additional context**
Add any other context or screenshots about the feature request here.

A similar issue is raised here by other people: https://github.com/aws/sagemaker-python-sdk/issues/1096

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。