huggingface / huggingface/tokenizers

Cannot import name '__version__' from 'tokenizers.tokenizers'

Open
#1,741 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

My admittedly complex setup for ComfyUI has been on the fritz. This is just the latest error.

But I've tried everything under the sun.

I uninstalled Tokenizers/Transformers and installed with `pip install transformers -U`. Nope. (Purged cache between installs, too).

I sourced the Git Repo, built it in Cargo, installed the compiled bindings to my virtual instance via `poetry run pip install -e path/to/bindings`. Nope.

Then I tried to add __version__='0.21.0' to the init.py and still a no-go.

I tried different versions of Tokenizers/Transformers, too.

The full error logs if for some reason it matters:

```
Traceback (most recent call last):
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1863, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/local/mise/installs/python/3.11.9/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/models/clip/tokenization_clip.py", line 25, in
from ...tokenization_utils import AddedToken, PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 27, in
from .tokenization_utils_base import (
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 91, in
from tokenizers import AddedToken
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/__init__.py", line 78, in
from .tokenizers import (
ImportError: cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/main.py", line 136, in
import execution
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/execution.py", line 13, in
import nodes
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/nodes.py", line 22, in
import comfy.diffusers_load
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/diffusers_load.py", line 3, in
import comfy.sd
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd.py", line 24, in
from . import model_detection
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/model_detection.py", line 1, in
import comfy.supported_models
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/supported_models.py", line 5, in
from . import sd1_clip
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
File "", line 1229, in _handle_fromlist
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1852, in __getattr__
value = getattr(module, name)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1851, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1865, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip.tokenization_clip because of the following error (look up to see its traceback):
cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)
```

And I'm using a Mac M1. Any help is much appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.