aws / aws/amazon-sagemaker-examples
[Bug Report] Training process from domain-adaption-finetuning-gpt-j-6b fails with ml.p3.2xlarge instance
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/domain-adaption-finetuning-gpt-j-6b.ipynb)
**Describe the bug**
Training process fails with ml.p3.2xlarge instance. The region is us-west-2. That was the largest instance I could use at my organisation.
**To reproduce**
Run the notebooks from the link with ml.g5.12xlarge changed to ml.p3.2xlarge instance.
**Logs**
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /opt/ml/code/transfer_learning.py:77 in │
│ │
│ 74 │ args, unknown = args_utils._parse_args() │
│ 75 │ logging.info(f"Running training scripts with arguments: {args}.") │
│ 76 │ logging.info(f"Ignoring unrecognized arguments: {unknown}.") │
│ ❱ 77 │ run_with_args(args) │
│ 78 │
│ │
│ /opt/ml/code/transfer_learning.py:19 in run_with_args │
│ │
│ 16 def run_with_args(args) -> None:
These are last logs from the CloudWatch. It looks like there is a problem with running transfer_learning.py entry point. I would appreciate any help.
Contributor guide
Research direction
Start with the domain-adaption-finetuning-gpt-j-6b.ipynb notebook and reproduce the run in us-west-2 using ml.p3.2xlarge instead of ml.g5.12xlarge. Inspect the transfer_learning.py entry point and the available CloudWatch traceback; done means the instance-specific training failure is understood and the notebook behavior or requirements are clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, machine-learning, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100