huggingface / huggingface/sentence-transformers
Multilingual models
- Dominant language
- Python
- Stars
- 19.1k
- Forks
- 2.9k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 61
Description
Hi there and thanks for the awesome work!
I'm currently working on a project that has some pre-tokenized content ids using the `bert-base-multilingual-cased`. Ideally, I'd like to just get the sentence encoding from the input ids (which I think I can do with a minor tweak in the code.
However, looking at the `distiluse-base-multilingual-cased` tokenisation results and the one done through the DistilBertTokenizer, the token ids of the the same output seem quite different:
```
from transformers import DistilBertTokenizer
from sentence_transformers import SentenceTransformer
sent_model = SentenceTransformer("distiluse-base-multilingual-cased")
sent_model.tokenize("Hello, how are you?")
[31178, 117, 14796, 10301, 13028, 136]
distil_tok = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
distil_tok.encode("Hello, how are you?")
[101, 61694, 10133, 117, 14796, 10301, 13028, 136, 102]
```
Am I wrong to assume that these tokenisation results should be the same?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at SentenceTransformer.tokenize and DistilBertTokenizer.encode using the shown example, and compare how each handles the model tokenizer and special tokens. Done means establishing whether the token IDs should match and documenting a clear answer or a precisely scoped fix if they should.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100