brain-score / brain-score/language
`digest_text` Huggingface tokenization logic may insert extraneous BOS/EOS tokens
- Dominant language
- Jupyter Notebook
- Stars
- 45
- Forks
- 21
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 1
Description
I'm assuming `digest_text` is designed to accept a list of strings corresponding to sentence sub-parts -- this matches what Martin seems to expect (cf. #54 ).
Note that many Huggingface tokenizers insert beginning-of-sentence / end-of-sentence (BOS/EOS) tokens when tokenizing. See e.g. RoBERTa:
```python
In [14]: tokenizer = transformers.AutoTokenizer.from_pretrained("xlm-roberta-base")
In [15]: [tokenizer.decode(sent) for sent in tokenizer(["the quick", "brown fox jumped", "over"]).input_ids]
Out[15]: [' the quick', ' brown fox jumped', ' over']
```
The current implementation seems like it will get tripped up by this: https://github.com/brain-score/language/blob/733df66736b4e63d1df013f72cbe18c95a32ce36/brainscore_language/models/huggingface.py#L69
If you know that inputs in your list are all part of the same sentence, you'll want to manually remove these, or maybe consider using `add_special_tokens=False` for subsequent tokenizer calls. See `tokenizer.bos_token` / `tokenizer.bos_token_id` and so on, or `tokenizer.all_special_tokens`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read digest_text in brainscore_language/models/huggingface.py around line 69, then compare its handling of list elements with the tokenizer behavior shown in the issue. Verify that tokenization of sentence sub-parts does not introduce unwanted BOS/EOS tokens between parts, and confirm the behavior with a focused regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100