huggingface / huggingface/tokenizers

Rust Issue on Unix in 0.21.0 Version

Open
#1,694 1 comment 7 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hi,

While installing the tokenizers library after 0.21.0 version release, the installation is failing in Unix. (Python 3.9.16)

It is looking for a Rust installation.

12:27:55 Looking in indexes: https://pypi.org/simple
12:27:55 Collecting tokenizers (from -r requirements.txt (line 1))
12:27:56 Downloading https:///pypi/pypi-/tokenizers-0.21.0.tar.gz (343 kB)
12:27:56 Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Checking for Rust toolchain....
Cargo, the Rust package manager, is not installed or is not on PATH.
This package requires Rust and Cargo to compile extensions. Install it through
the system's package manager or via https://rustup.rs/

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

When I revert to 0.20.3 , it works fine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.