huggingface / huggingface/setfit
Best Approach for adding new vocabulary?
Open
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
What would be the best approach for adding new vocab to the tokenizer before training the model? I tried accessing the tokenizer directly but realized there would be no way to resize the token_embeddings in the underlying SentenceTransformer object model.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.