huggingface / huggingface/tokenizers
Slow compile times
Open
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Compiling `tokenizers` takes around 35 seconds in release mode.
Most of this (25s) is spend compiling `onig_sys`.
An alternative to using `onig_sys` would be appreciated. This crate already uses the `regex-*` ecosystem, so if that could be used, it would be a big win for compile times and binary sizes.
Otherwise, if you point me to where other regex libraries fall down, I'm happy to look into alternatives.
Contributor guide
Assessment
This issue has not been assessed yet.