aws / aws/sagemaker-experiments
Tracker.load() does not work when using sagemaker pipeline
- Dominant language
- Python
- Stars
- 129
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Create Experiment and Trial `smexperiment`. And configure `PipelineExperimentConfig`.
Run training Job using sagemaker pipeline. (When without sagemaker pipeline, this bug did not occur)
In training script, `Tracker.load()` return exception about this
```
Traceback (most recent call last):
File "train.py", line 133, in
main()
File "train.py", line 66, in main
tracker = Tracker.load()
File "/miniconda3/lib/python3.8/site-packages/smexperiments/tracker.py", line 161, in load
_ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session),
AttributeError: 'NoneType' object has no attribute 'trial_component_name'
```
Maybe [this function](https://github.com/aws/sagemaker-experiments/blob/52800e4ecd22a6ac43aa64086658a5654c11413c/src/smexperiments/_environment.py#L68) return `None`.
But `TrialComponentEnvironment.source_arn` is defined.
So, I guess is [this line](https://github.com/aws/sagemaker-experiments/blob/52800e4ecd22a6ac43aa64086658a5654c11413c/src/smexperiments/_environment.py#L81) wrong?
Because `environment["TRAINING_JOB_ARN"]` contains uppercase when using sagemaker pipeline.
Sorry for my poor English.
**To Reproduce**
1. configure Experiment, Trial and PipelineExperimentConfig
2. run training job using pipeline
3. `Tracker.load()` in training script
**Expected behavior**
`Tracker.load()` load trial_component in pipeline
**Environment:**
Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
Framework Version:
Python Version: 3.9.11
CPU or GPU:CPU
Python SDK Version:
- sagemaker==2.116.0
- sagemaker-experiments==0.1.39 (in image: 0.1.41)
Are you using a custom image: yes
Contributor guide
Assessment
This issue has not been assessed yet.