aws / aws/amazon-sagemaker-examples

[Bug Report] No model output after training

Open
#4,372 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/51cbf4a77c98dc9b74fde6a8c47db0dad40fb910/training/distributed_training/pytorch/model_parallel/gpt-j/smp-train-gptj-sharded-data-parallel-tp.ipynb

**Describe the bug**
This notebook is to fine tune gpt-j model. I used one ml.p4d.24xlarge instance in us-east-1 and other options are default.
The training process is very smooth, after 18 minutes, training process complete. However I can not find model output in S3 folder "smp-tensorparallel-outputdir", I can access this S3 folder, but it is empty.

Hope you can help. thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.