aws / aws/amazon-sagemaker-examples
[Bug Report] No model output after training
Open
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/51cbf4a77c98dc9b74fde6a8c47db0dad40fb910/training/distributed_training/pytorch/model_parallel/gpt-j/smp-train-gptj-sharded-data-parallel-tp.ipynb
**Describe the bug**
This notebook is to fine tune gpt-j model. I used one ml.p4d.24xlarge instance in us-east-1 and other options are default.
The training process is very smooth, after 18 minutes, training process complete. However I can not find model output in S3 folder "smp-tensorparallel-outputdir", I can access this S3 folder, but it is empty.
Hope you can help. thanks.
Contributor guide
Assessment
This issue has not been assessed yet.