huggingface / huggingface/tokenizers
Tokenizer Training Errors: pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: TryFromIntError(())
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Hi, I was trying to train a Unigram tokenizer with DNA sequence data. And this is the code I use to train:
```
print("Unigram tokenizer")
tokenizer = Tokenizer(Unigram())
special_tokens = ["", "", "", "", "", "", ""]
trainer = UnigramTrainer(special_tokens=special_tokens, vocab_size=Unigram_vocab_size, unk_token="", show_progress=True)
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
tokenizer.save(join(output_folder, "SentencePiece_"+ str(vocab_size) + ".json"))
```
At first, I encountered this error:
(base) [shfa523g@login1.barnard slurm_stderr]$ cat slurm-12831381.out
thread '' panicked at /home/runner/work/tokenizers/tokenizers/tokenizers/src/models/unigram/trainer.rs:228:53:
called `Result::unwrap()` on an `Err` value: Internal
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/shfa523g/Chi_Internship/tokenizers_scripts/generate_tokenizers_parallel.py", line 90, in
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: Internal
Then I found a similar issue description and did as what it said, the issue is here:
https://github.com/huggingface/tokenizers/issues/821#issuecomment-958144662
After I tried what it says, it gives me a new issue:
(base) [shfa523g@login1.barnard slurm_stderr]$ cat slurm-12861633.out
thread '' panicked at /home/shfa523g/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esaxx-rs-0.1.10/src/esa.rs:70:50:
called `Result::unwrap()` on an `Err` value: TryFromIntError(())
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/shfa523g/Chi_Internship/tokenizers_scripts/generate_tokenizers_parallel.py", line 90, in
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: TryFromIntError(())
I asked ChatGPT, it says it might be the sequences it took are too long, then I reduce it to a very small number but the same error keeps happening.
Please take a look, thank you very much!
Contributor guide
Assessment
This issue has not been assessed yet.