aws / aws/sagemaker-python-sdk
need to be able to set return data size for each request in batch transform
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 32
Description
**Describe the feature you'd like**
I'm running a NLP inference job to get sentence embedding vector for each record. Each record is less than 512 words only and the returned vector has 768 floats. Even if I set max_payload to 1 and max_concurrent_transforms to 1, I still got:
**io.netty.handler.codec.CorruptedFrameException: Message size exceed limit: 113038147**
I know there is a 5MB limit for endpoint inference. I'm relatively sure that this is caused by the size limit of return results of each request from batch transform, though I'm not able to find it in the doc.
I hope you can
1. Clarify the size limit for each request for batch transform in the doc.
2. Mimicking max_payload, expose a parameter to let user control the return data size for each request for batch transform.
**How would this feature be used? Please describe.**
Add a max_return_payload parameter to model.transformer like below.
tfm = model.transformer(instance_count=1,
instance_type='ml.p3.2xlarge',
accept='text/json',
assemble_with='Line',
output_path=batch_out,
strategy='MultiRecord',
max_payload=3,
**max_return_payload=30**)
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
Use strategy='SingleRecord' can bypass the issue. But it is significant slower as it's not making use of parallelism. Setting max_concurrent_transform to a larger value can make use of parallelism. But it could cause problem for code that's not designed for concurrency.
**Additional context**
Add any other context or screenshots about the feature request here.
A similar issue is raised here by other people: https://github.com/aws/sagemaker-python-sdk/issues/1096
Contributor guide
Assessment
This issue has not been assessed yet.