aws / aws/sagemaker-huggingface-inference-toolkit

SageMaker fails because Conversation object is not found

Open
#129 15 comments 14 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

We are hosting a model in SageMaker and today we observed the following error in our logs when the model was being relaunched in the instance:

```
ImportError: cannot import name 'Conversation' from 'transformers.pipelines' (/opt/conda/lib/python3.9/site-packages/transformers/pipelines/__init__.py)
```

I found someone reported a similar issue in https://discuss.huggingface.co/t/cannot-import-conversation-from-transformers-utils-py/91556. When digging into the changes in the `transformers` dependency I found this change regarding the `Conversation` object: https://github.com/huggingface/transformers/pull/31165

Based in our logs, the last time the model was successfully relaunched in our SageMaker infrastructure (2 days ago), the version of the package that was downloaded was `transformers-4.41.2-py3-none-any.whl`, but when the error started to be observed, the downloaded version was `transformers-4.42.1-py3-none-any.whl`.

According to the pull request mentioned above, the change is effective as per version `4.42`, which was released 18 hours ago at the moment of writing this issue.

I think a change is needed in `src/sagemaker_huggingface_inference_toolkit/transformers_utils.py` to reflect the new structure of the code in the `transformers` dependency or pin the dependency version to version `4.41.2` to prevent the issue in the future.

In our case, we are updating our `requirements.txt` file to pin the version `4.41.2`, but other users might not be aware of what is happening, therefore they are not aware of this fix.

Contributor guide

Open the contributing guide

Research direction

Start by reading src/sagemaker_huggingface_inference_toolkit/transformers_utils.py and compare its Conversation import with the transformers 4.42.1 package structure. Reproduce the SageMaker model relaunch using the reported dependency versions; done means the toolkit loads without the ImportError while supporting the changed transformers dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.