aws / aws/sagemaker-experiments

Tracker.load() does not work when using sagemaker pipeline

Open
#174 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
129
Forks
36
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Create Experiment and Trial `smexperiment`. And configure `PipelineExperimentConfig`.
Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script, `Tracker.load()` return exception about this
```
Traceback (most recent call last):
File "train.py", line 133, in
main()
File "train.py", line 66, in main
tracker = Tracker.load()
File "/miniconda3/lib/python3.8/site-packages/smexperiments/tracker.py", line 161, in load
_ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session),
AttributeError: 'NoneType' object has no attribute 'trial_component_name'
```
Maybe [this function](https://github.com/aws/sagemaker-experiments/blob/52800e4ecd22a6ac43aa64086658a5654c11413c/src/smexperiments/_environment.py#L68) return `None`.
But `TrialComponentEnvironment.source_arn` is defined.
So, I guess is [this line](https://github.com/aws/sagemaker-experiments/blob/52800e4ecd22a6ac43aa64086658a5654c11413c/src/smexperiments/_environment.py#L81) wrong?
Because `environment["TRAINING_JOB_ARN"]` contains uppercase when using sagemaker pipeline.

Sorry for my poor English.

**To Reproduce**
1. configure Experiment, Trial and PipelineExperimentConfig
2. run training job using pipeline
3. `Tracker.load()` in training script

**Expected behavior**
`Tracker.load()` load trial_component in pipeline

**Environment:**
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:
- sagemaker==2.116.0
- sagemaker-experiments==0.1.39 (in image: 0.1.41)

Are you using a custom image: yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.