huggingface / huggingface/tokenizers
Cannot import name '__version__' from 'tokenizers.tokenizers'
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
My admittedly complex setup for ComfyUI has been on the fritz. This is just the latest error.
But I've tried everything under the sun.
I uninstalled Tokenizers/Transformers and installed with `pip install transformers -U`. Nope. (Purged cache between installs, too).
I sourced the Git Repo, built it in Cargo, installed the compiled bindings to my virtual instance via `poetry run pip install -e path/to/bindings`. Nope.
Then I tried to add __version__='0.21.0' to the init.py and still a no-go.
I tried different versions of Tokenizers/Transformers, too.
The full error logs if for some reason it matters:
```
Traceback (most recent call last):
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1863, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/local/mise/installs/python/3.11.9/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/models/clip/tokenization_clip.py", line 25, in
from ...tokenization_utils import AddedToken, PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 27, in
from .tokenization_utils_base import (
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 91, in
from tokenizers import AddedToken
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/__init__.py", line 78, in
from .tokenizers import (
ImportError: cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/main.py", line 136, in
import execution
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/execution.py", line 13, in
import nodes
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/nodes.py", line 22, in
import comfy.diffusers_load
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/diffusers_load.py", line 3, in
import comfy.sd
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd.py", line 24, in
from . import model_detection
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/model_detection.py", line 1, in
import comfy.supported_models
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/supported_models.py", line 5, in
from . import sd1_clip
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
File "", line 1229, in _handle_fromlist
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1852, in __getattr__
value = getattr(module, name)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1851, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1865, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip.tokenization_clip because of the following error (look up to see its traceback):
cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)
```
And I'm using a Mac M1. Any help is much appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.