huggingface / huggingface/tokenizers

Slow compile times

Open
#1,730 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Compiling `tokenizers` takes around 35 seconds in release mode.
Most of this (25s) is spend compiling `onig_sys`.

An alternative to using `onig_sys` would be appreciated. This crate already uses the `regex-*` ecosystem, so if that could be used, it would be a big win for compile times and binary sizes.

Otherwise, if you point me to where other regex libraries fall down, I'm happy to look into alternatives.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.