huggingface / huggingface/tokenizers

ERROR occurs when running "tokenizer._tokenizer.model.clear_cache()"

Open
#1,738 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

I met an OOM problem when using BertTokenizer as https://github.com/huggingface/tokenizers/issues/1539 reports.

Then I use tokenizer._tokenizer.model.clear_cache() or tokenizer._tokenizer.model._clear_cache() to clear cache.

However, I met an error: AttributeError: 'tokenizers.models.WordPiece' object has no attribute 'clear_cache', could anyone tell me how to fix it?

In the source code, It seems like clear_cache only supports BPE and Unigram tokenizer, not wordpiece tokenizer, is it the reason? if it is, could anyone give me some advice to fix this problem?

environment:
run on linux with only cpu
tokenizers==0.21.0
transformers==4.49.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.