aws / aws/aws-step-functions-data-science-sdk-python

adding tags to a Sagemaker estimator in the training step does not seem to be supported

Open
#200 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
299
Forks
84
PR merge metrics
No merged PRs in 30d

Description

Extract from the workbook "machine_learning_workflow_abalone.ipynb"
When adding tags in the following estimator :

mes_tags = [{'key': 'cart', 'value': 'dataengineering'}]

xgb = sagemaker.estimator.Estimator(
image_uris.retrieve("xgboost", region, "1.2-1"),
sagemaker_execution_role,
train_instance_count=1,
train_instance_type="ml.m4.4xlarge",
train_volume_size=5,
output_path=bucket_path + "/" + prefix + "/single-xgboost",
base_job_name=base_job_name,
tags=mes_tags,
sagemaker_session=session,
)
No error when creating the sagemaker.estimator object

The workflow creation fails
When running the command (later in the notebook):
workflow.create()
I got the exception :
"InvalidDefinition: An error occurred (InvalidDefinition) when calling the CreateStateMachine operation: Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The field "key" is not supported by Step Functions at /States/Train Step/Parameters"

Which is clearly related to the tags I previously added.
Apparently, adding tags to a Sagemaker estimator in the training step does not seem to be supported by the current version of the SDK.

To reproduce
You can comment "tags=mes_tags" and re-rerun the notebook and the state machine is created without any errors.

Logs
Only the stack trace in the notebook

Contributor guide

Open the contributing guide

Research direction

Start with machine_learning_workflow_abalone.ipynb and compare workflow.create() with and without the estimator's tags argument. Inspect the generated Train Step Parameters around the reported key validation error and the estimator/workflow entry points involved. Done means the tagged training workflow is accepted by Step Functions without the reported InvalidDefinition error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.