huggingface / huggingface/tokenizers
ERROR occurs when running "tokenizer._tokenizer.model.clear_cache()"
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
I met an OOM problem when using BertTokenizer as https://github.com/huggingface/tokenizers/issues/1539 reports.
Then I use tokenizer._tokenizer.model.clear_cache() or tokenizer._tokenizer.model._clear_cache() to clear cache.
However, I met an error: AttributeError: 'tokenizers.models.WordPiece' object has no attribute 'clear_cache', could anyone tell me how to fix it?
In the source code, It seems like clear_cache only supports BPE and Unigram tokenizer, not wordpiece tokenizer, is it the reason? if it is, could anyone give me some advice to fix this problem?
environment:
run on linux with only cpu
tokenizers==0.21.0
transformers==4.49.0
Contributor guide
Assessment
This issue has not been assessed yet.