huggingface / huggingface/sentence-transformers

Multilingual models

Open
#182 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.1k
Forks
2.9k
Avg merge
1d 19h
Merged PRs (30d)
61

Description

Hi there and thanks for the awesome work!
I'm currently working on a project that has some pre-tokenized content ids using the `bert-base-multilingual-cased`. Ideally, I'd like to just get the sentence encoding from the input ids (which I think I can do with a minor tweak in the code.
However, looking at the `distiluse-base-multilingual-cased` tokenisation results and the one done through the DistilBertTokenizer, the token ids of the the same output seem quite different:

```
from transformers import DistilBertTokenizer
from sentence_transformers import SentenceTransformer
sent_model = SentenceTransformer("distiluse-base-multilingual-cased")
sent_model.tokenize("Hello, how are you?")
[31178, 117, 14796, 10301, 13028, 136]
distil_tok = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
distil_tok.encode("Hello, how are you?")
[101, 61694, 10133, 117, 14796, 10301, 13028, 136, 102]
```
Am I wrong to assume that these tokenisation results should be the same?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at SentenceTransformer.tokenize and DistilBertTokenizer.encode using the shown example, and compare how each handles the model tokenizer and special tokens. Done means establishing whether the token IDs should match and documenting a clear answer or a precisely scoped fix if they should.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.