aws / aws/amazon-sagemaker-examples

[Bug Report] Training process from domain-adaption-finetuning-gpt-j-6b fails with ml.p3.2xlarge instance

Open
#4,347 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/domain-adaption-finetuning-gpt-j-6b.ipynb)

**Describe the bug**
Training process fails with ml.p3.2xlarge instance. The region is us-west-2. That was the largest instance I could use at my organisation.

**To reproduce**
Run the notebooks from the link with ml.g5.12xlarge changed to ml.p3.2xlarge instance.

**Logs**
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /opt/ml/code/transfer_learning.py:77 in │
│ │
│ 74 │ args, unknown = args_utils._parse_args() │
│ 75 │ logging.info(f"Running training scripts with arguments: {args}.") │
│ 76 │ logging.info(f"Ignoring unrecognized arguments: {unknown}.") │
│ ❱ 77 │ run_with_args(args) │
│ 78 │
│ │
│ /opt/ml/code/transfer_learning.py:19 in run_with_args │
│ │
│ 16 def run_with_args(args) -> None:

These are last logs from the CloudWatch. It looks like there is a problem with running transfer_learning.py entry point. I would appreciate any help.

Contributor guide

Open the contributing guide

Research direction

Start with the domain-adaption-finetuning-gpt-j-6b.ipynb notebook and reproduce the run in us-west-2 using ml.p3.2xlarge instead of ml.g5.12xlarge. Inspect the transfer_learning.py entry point and the available CloudWatch traceback; done means the instance-specific training failure is understood and the notebook behavior or requirements are clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, machine-learning, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.