aws / aws/amazon-sagemaker-clarify

Carify job fails in spark mode

Open
#127 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
75
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Thanks for this project. For my project, I'd need to configure some elements of the clarify processing and it would require respective [Docker Files](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-processing-job-configure-container.html) available for modification. More concretely, I am facing timeouts in the endpoint calls due to a very high max batch size/max payload and a slow model, but only when apache spark integration is used, i.e. `instance_count > 1`. In that case, the max payload is for some reason much higher than when spark integration is disabled, leading to longer response times for a batch. Choosing more or a bigger or more powerful instance in the endpoint does not solve the problem.

Can you open-source the Dockerfiles? This would be very beneficial.

In addition, `sagemaker.clarify.SageMakerClarifyProcessor()` should accept an optional `image_uri` argument so I can supply my custom image, but that I can also solve myself by forking the sagemaker sdk and create a PR

Contributor guide

Open the contributing guide

Research direction

Start at the sagemaker.clarify.SageMakerClarifyProcessor() entry point and locate the Docker build definitions for Clarify processing. Confirm how Spark mode selects its image and how endpoint batch limits are configured; done means the Dockerfiles are available for modification and the processor accepts an optional image_uri for a custom image.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, spark
Domain
api, devops, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.