aws / aws/amazon-sagemaker-examples

GPT-4ALL-J on Inferentia2

Open
#4,128 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/inference/generativeai/llm-workshop/lab8-Inferentia2-gpt4all-j/inferentia2-llm-GPT4allJ.ipynb

**Describe the bug**
Keep getting the following warning:
- Your model is not compiled. Please compile your model before using Inferentia.

and this error:
- ClientError: An error occurred (ValidationException) when calling the CreateEndpointConfig operation: Instance type ml.inf2.8xlarge is not supported for the chosen region. Please choose some other instance type.

**To reproduce**
Same steps as stated in notebook

I have already set my region to us-east-1 for all services, yet I still encounter the error for Inferentia2 instance

Contributor guide

Open the contributing guide

Research direction

Start with the linked inferentia2-llm-GPT4allJ.ipynb and inspect the deployment cells that compile the model and create the endpoint configuration. Reproduce the reported warning and ValidationException in us-east-1; done means the notebook’s Inferentia2 deployment completes without these errors or clearly identifies the unsupported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.