aws / aws/amazon-sagemaker-examples
Training job does not create model.tar.gz
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/smp-train-gptj-sharded-data-parallel-tp.ipynb
**Describe the bug**
Running the training job without any change does not create any model.tar.gz
Hence it cannot be deployed.
If we set "save_final_full_model": 1 ( default =0), then it creates a model.tar.gz which has the following two files:
fullmodel.pt
user_content_fullmodel.pt
When we try to deploy this it gives an error saying that no config.json has been found.
**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.
**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
Contributor guide
Research direction
Start with the linked notebook, smp-train-gptj-sharded-data-parallel-tp.ipynb, and run the training job without changes to inspect its output artifacts. Compare this with the save_final_full_model setting and the deployment error; done means the training job creates a deployable model.tar.gz containing the required config.json.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100