sentencepiece is DEAD and needs to be removed as a dependency

### Expected Behavior

Install ComfyUI on an updated (modern/2025) linux environment.

### Actual Behavior

ComfyUI installation fails due to a broken dependency on sentencepiece, which is dead and no longer under development.

### Steps to Reproduce

Install ComfyUI on a 2025 or late 2024 linux distribution. Installation fails because python 3.13 is the default and often the only option.

### Debug Logs

```powershell
Building wheels for collected packages: sentencepiece
Building wheel for sentencepiece (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for sentencepiece (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [147 lines of output]
/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'test_suite'
warnings.warn(msg)
/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

********************************************************************************
Please consider removing the following classifiers in favor of a SPDX license expression:

License :: OSI Approved :: Apache Software License

See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
********************************************************************************

!!
self._finalize_license_expression()
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-313/sentencepiece
copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
running build_ext
Package sentencepiece was not found in the pkg-config search path.
Perhaps you should add the directory containing `sentencepiece.pc'
to the PKG_CONFIG_PATH environment variable
Package 'sentencepiece' not found
CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
Compatibility with CMake < 3.10 will be removed from a future version of
CMake.

Update the VERSION argument value. Or, use the ... syntax
to tell CMake that the project requires at least but has been updated
to work with policies introduced by or earlier.


-- VERSION: 0.2.0
-- The C compiler identification is GNU 15.0.1
-- The CXX compiler identification is unknown
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/sbin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at CMakeLists.txt:23 (project):
No CMAKE_CXX_COMPILER could be found.

Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.


-- Configuring incomplete, errors occurred!
gmake: Makefile: No such file or directory
gmake: *** No rule to make target 'Makefile'. Stop.
Traceback (most recent call last):
File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
~~~~^^
File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
metadata_directory)
^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 435, in build_wheel
return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 423, in _build
return self._build_with_temp_dir(
~~~~~~~~~~~~~~~~~~~~~~~~~^
cmd,
^^^^
...<3 lines>...
self._arbitrary_args(config_settings),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
self.run_setup()
~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 512, in run_setup
super().run_setup(setup_script=setup_script)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup
exec(code, locals())
~~~~^^^^^^^^^^^^^^^^
File "", line 169, in
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
return run_commands(dist)
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
dist.run_commands()
~~~~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
self.run_command(cmd)
~~~~~~~~~~~~~~~~^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
self.run_command("build")
~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 99, in run
_build_ext.run(self)
~~~~~~~~~~~~~~^^^^^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
self.build_extensions()
~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
self._build_extensions_serial()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
self.build_extension(ext)
~~~~~~~~~~~~~~~~~~~~^^^^^
File "", line 87, in build_extension
File "/usr/lib64/python3.13/subprocess.py", line 421, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.2.0']' returned non-zero exit status 2.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for sentencepiece
Failed to build sentencepiece

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (sentencepiece)
```

### Other

As the title says, sentencepiece is no longer being updated, and is falling behind. ComfyUI devs need to find an alternative and replace it in the dependencies. Recommend using tokenizers instead. The issue was closed (https://github.com/comfyanonymous/ComfyUI/issues/6169) but should not have been. This is a *real* issue, and the devs need to sort it out.

🔷 Hugging Face tokenizers
Benefits:

Performance: Extremely fast due to Rust backend with Python bindings.

Flexibility: Offers a wide range of tokenization algorithms (BPE, WordPiece, Unigram).

Customizability: Fine-grained control over preprocessing, postprocessing, normalization, etc.

Integration: Native support in Hugging Face Transformers ecosystem.

Python 3.13 Support: Actively maintained and compatible with latest Python versions.

Drawbacks:

Complexity: API is powerful but may be overwhelming for simple use cases.

Lesser Standardization: Not as widely used in traditional NLP pipelines as SentencePiece.

Binary Files: Requires compilation or pre-built wheels (minor concern, but relevant in limited environments).

🔷 Google sentencepiece
Benefits:

Language-Agnostic: Designed to work on raw text without language-specific rules (especially useful for multilingual applications).

Simplicity: Easy CLI and Python API for training and encoding/decoding.

Proven Track Record: Used in production at Google, supports models like T5, mT5, and XLNet.

Unigram Model: Provides robust subword segmentation based on a probabilistic model.

Drawbacks:

Performance: Slower than tokenizers, especially in large-scale or real-time scenarios.

Python 3.13 Incompatibility: As of now, it does not officially support Python 3.13.

Less Customizable: Harder to fine-tune preprocessing or add custom behavior compared to tokenizers.