aws / aws/amazon-sagemaker-examples

Training job does not create model.tar.gz

Open
#4,346 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/smp-train-gptj-sharded-data-parallel-tp.ipynb

**Describe the bug**
Running the training job without any change does not create any model.tar.gz
Hence it cannot be deployed.
If we set "save_final_full_model": 1 ( default =0), then it creates a model.tar.gz which has the following two files:
fullmodel.pt
user_content_fullmodel.pt

When we try to deploy this it gives an error saying that no config.json has been found.

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.

Contributor guide

Open the contributing guide

Research direction

Start with the linked notebook, smp-train-gptj-sharded-data-parallel-tp.ipynb, and run the training job without changes to inspect its output artifacts. Compare this with the save_final_full_model setting and the deployment error; done means the training job creates a deployable model.tar.gz containing the required config.json.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.