aws / aws/amazon-sagemaker-examples
No logs shown after training completion
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
using my custom algorithm for model training and After running
`estimator.fit({'training':'s3://abc/xxx/train.csv','validation':'s3://abc/xxx/val.csv'})`
i get the below message:-
```
2020-05-24 21:08:52 Starting - Starting the training job...
2020-05-24 21:08:54 Starting - Launching requested ML instances...
2020-05-24 21:10:07 Starting - Preparing the instances for training............
2020-05-24 21:12:17 Downloading - Downloading input data
2020-05-24 21:12:17 Training - Downloading the training image.........
2020-05-24 21:14:16 Uploading - Uploading generated training model...
2020-05-24 21:14:22 Completed - Training job completed
..Training seconds: 143
Billable seconds: 43
Managed Spot Training savings: 69.9%
```
now this really fishy as i am training a bert model for 40 epochs which just cant be done in this much of time. Also , i dont see any logs whatsoever in the cloudwatch.
what is really happening here ? can someone please help me out?
Contributor guide
Assessment
This issue has not been assessed yet.