huggingface / huggingface/tokenizers

Tokenizer Training Errors: pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: TryFromIntError(())

Open
#1,698 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

Hi, I was trying to train a Unigram tokenizer with DNA sequence data. And this is the code I use to train:

```
print("Unigram tokenizer")
tokenizer = Tokenizer(Unigram())
special_tokens = ["", "", "", "", "", "", ""]
trainer = UnigramTrainer(special_tokens=special_tokens, vocab_size=Unigram_vocab_size, unk_token="", show_progress=True)
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
tokenizer.save(join(output_folder, "SentencePiece_"+ str(vocab_size) + ".json"))
```

At first, I encountered this error:
(base) [shfa523g@login1.barnard slurm_stderr]$ cat slurm-12831381.out
thread '' panicked at /home/runner/work/tokenizers/tokenizers/tokenizers/src/models/unigram/trainer.rs:228:53:
called `Result::unwrap()` on an `Err` value: Internal
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/shfa523g/Chi_Internship/tokenizers_scripts/generate_tokenizers_parallel.py", line 90, in
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: Internal

Then I found a similar issue description and did as what it said, the issue is here:
https://github.com/huggingface/tokenizers/issues/821#issuecomment-958144662

After I tried what it says, it gives me a new issue:
(base) [shfa523g@login1.barnard slurm_stderr]$ cat slurm-12861633.out
thread '' panicked at /home/shfa523g/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esaxx-rs-0.1.10/src/esa.rs:70:50:
called `Result::unwrap()` on an `Err` value: TryFromIntError(())
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/shfa523g/Chi_Internship/tokenizers_scripts/generate_tokenizers_parallel.py", line 90, in
tokenizer.train_from_iterator(iterator=all_seqs, trainer=trainer)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: TryFromIntError(())

I asked ChatGPT, it says it might be the sequences it took are too long, then I reduce it to a very small number but the same error keeps happening.

Please take a look, thank you very much!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.