aws / aws/amazon-sagemaker-examples

[Bug Report] RuntimeError when running instruction fine-tuning on mistral 7b, Sagemaker Jumpstart

Open
#4,649 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/mistral-7b-instruction-domain-adaptation-finetuning.ipynb

**Describe the bug**
I get an error when I run the training step for instruction fine-tuning in this notebook. The training job starts properly, but after ~10min it fails and raises: `ErrorMessage "raise RuntimeError( RuntimeError: Could not find response key [1, 32002] in token IDs tensor([ 1, 20811, 349, ..., 302, 15637, 266])`

**To reproduce**
- Upload the notebook in a Sagemaker Notebook
- Run every cell, the error appears when running the instruction-fine tuning training job (**1.3 Starting Training** section)

**Logs**
Attaching some screenshots of the logs

![Screenshot 2024-05-03 at 16 45 43](https://github.com/aws/amazon-sagemaker-examples/assets/43200045/a3b511a0-8dd0-4489-95da-326192f7bdce)

![Screenshot 2024-05-03 at 16 47 10](https://github.com/aws/amazon-sagemaker-examples/assets/43200045/ccea49b4-5e39-4c59-bb5d-85c9d8ce96d6)

Any idea on how to fix this ?

Contributor guide

Open the contributing guide

Research direction

Open the linked mistral-7b-instruction-domain-adaptation-finetuning.ipynb in a SageMaker Notebook and reproduce the failure in the “1.3 Starting Training” section. Start by reviewing the training-job logs around the reported response-key error; done means the instruction fine-tuning job completes successfully without the RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.