aws / aws/amazon-sagemaker-examples
[Example Request] Training pipeline and using it later for inference
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Describe the use case example you want to see**
I was following this [example](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb) in which a pipeline is built by concatenating several steps. From the notebook is pretty clear how to build such a pipeline and the advantages. However, I'm missing how certain steps of the pipeline could be reused in a different python process at inference time.
Let's say I train this pipeline. I want to launch a Python script every hour that reads some CSV data from S3 and feeds it to the two first fitted steps of the pipeline (the sklearn preprocessor and the XGBoost model) to get the outputs and save them. I don't want to train these models again, just want to reuse the previously fitted steps. Would this be done just by retrieving the previously generated artifacts from a periodically launched container?. Or is there any SDK to achieve this?
**How would this example be used? Please describe.**
This would be used in a production system where predictions have to be stored for new incoming data in a periodic manner (1 hour for example)
**Describe which SageMaker services are involved**
Sagemaker Pipelines, Sagemaker Training jobs.
**Describe what other services (other than SageMaker) are involved***
Cloudwatch to monitor the metrics if possible
Contributor guide
Research direction
Start with sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb and review its training and batch-transform steps. Define an example showing how artifacts from the fitted preprocessing and XGBoost steps are reused by a periodically launched Python process, including the S3 input/output flow and the expected inference result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100