brain-score / brain-score/language

`digest_text` Huggingface tokenization logic may insert extraneous BOS/EOS tokens

Open
#56 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
45
Forks
21
Avg merge
1h 7m
Merged PRs (30d)
1

Description

I'm assuming `digest_text` is designed to accept a list of strings corresponding to sentence sub-parts -- this matches what Martin seems to expect (cf. #54 ).

Note that many Huggingface tokenizers insert beginning-of-sentence / end-of-sentence (BOS/EOS) tokens when tokenizing. See e.g. RoBERTa:

```python
In [14]: tokenizer = transformers.AutoTokenizer.from_pretrained("xlm-roberta-base")

In [15]: [tokenizer.decode(sent) for sent in tokenizer(["the quick", "brown fox jumped", "over"]).input_ids]
Out[15]: [' the quick', ' brown fox jumped', ' over']
```

The current implementation seems like it will get tripped up by this: https://github.com/brain-score/language/blob/733df66736b4e63d1df013f72cbe18c95a32ce36/brainscore_language/models/huggingface.py#L69

If you know that inputs in your list are all part of the same sentence, you'll want to manually remove these, or maybe consider using `add_special_tokens=False` for subsequent tokenizer calls. See `tokenizer.bos_token` / `tokenizer.bos_token_id` and so on, or `tokenizer.all_special_tokens`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read digest_text in brainscore_language/models/huggingface.py around line 69, then compare its handling of list elements with the tokenizer behavior shown in the issue. Verify that tokenization of sentence sub-parts does not introduce unwanted BOS/EOS tokens between parts, and confirm the behavior with a focused regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.