aws / aws/sagemaker-python-sdk

need to be able to set return data size for each request in batch transform

Đã đóng
#1,882 13 bình luận 1 reaction 1 người được giao Được @mollyheamazon nhận Xem trên GitHub
component: Inference APIs and Interfaces type: feature request
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**Describe the feature you'd like**
I'm running a NLP inference job to get sentence embedding vector for each record. Each record is less than 512 words only and the returned vector has 768 floats. Even if I set max_payload to 1 and max_concurrent_transforms to 1, I still got:
**io.netty.handler.codec.CorruptedFrameException: Message size exceed limit: 113038147**

I know there is a 5MB limit for endpoint inference. I'm relatively sure that this is caused by the size limit of return results of each request from batch transform, though I'm not able to find it in the doc.

I hope you can
1. Clarify the size limit for each request for batch transform in the doc.
2. Mimicking max_payload, expose a parameter to let user control the return data size for each request for batch transform.

**How would this feature be used? Please describe.**

Add a max_return_payload parameter to model.transformer like below.

tfm = model.transformer(instance_count=1,
instance_type='ml.p3.2xlarge',
accept='text/json',
assemble_with='Line',
output_path=batch_out,
strategy='MultiRecord',
max_payload=3,
**max_return_payload=30**)

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Use strategy='SingleRecord' can bypass the issue. But it is significant slower as it's not making use of parallelism. Setting max_concurrent_transform to a larger value can make use of parallelism. But it could cause problem for code that's not designed for concurrency.

**Additional context**
Add any other context or screenshots about the feature request here.

A similar issue is raised here by other people: https://github.com/aws/sagemaker-python-sdk/issues/1096

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.