aws / aws/aws-step-functions-data-science-sdk-python

timestamp mismatch when using code_location

Open
#55 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
299
Forks
84
PR merge metrics
No merged PRs in 30d

Description

HI,

When code_location is used in estimator of TrainingStep(), the uploaded s3 path and sagemaker_submit_directory timestamp do not match(about 400 ms).
This will cause the execution to fail.

In SageMaker training job, timestamp matches even if code_location is used.

S3 uploaded path
s3://my-bucket/model/sagemaker-xgboost-2020-06-10-06-29-37-910/source/sourcedir.tar.gz

sagemaker_submit_directory
"s3://my-bucket/model/sagemaker-xgboost-2020-06-10-06-29-38-323/source/sourcedir.tar.gz"

```
# Open Source distributed script mode
from sagemaker.session import s3_input, Session
from sagemaker.xgboost.estimator import XGBoost

boto_session = boto3.Session(region_name=region)
session = Session(boto_session=boto_session)

output_path = 's3://{}/{}'.format(bucket_name, 'model')

xgb_script_mode_estimator = XGBoost(
entry_point='xgboost.py',
source_dir='source',
framework_version='0.90-2', # Note: framework_version is mandatory
hyperparameters=hyperparams,
role=role,
train_instance_count=1,
train_instance_type='ml.m5.2xlarge',
code_location=output_path, # ← Cause a mismatch
output_path=output_path
)
```

Contributor guide

Open the contributing guide

Research direction

Start with the TrainingStep estimator and the code_location handling shown in the reproduction. Run the provided XGBoost example and compare the S3 uploaded path with sagemaker_submit_directory timestamps. Done means both paths use the same timestamp so the training execution succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.