huggingface / huggingface/olm-training

Generation config - "bos_token_id", "eos_token_id", "pad_token_id"

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
98
Forks
11
PR merge metrics
No merged PRs in 30d

Description

Thanks for making these models available. They're really useful.

Would it be possible to look at correcting the generation config for the gpt2 models? The model [config.json](https://huggingface.co/olm/olm-gpt2-latest/blob/main/config.json) has `"bos_token_id": 50256` and `"eos_token_id": 50256`, which I think is correct for the original gpt2, but not for the tokenizer used for this model. It think it might also be possible to set `"pad_token_id"`.

I noticed this when attempting to generate from an empty string:
```python
>>> from transformers import pipeline
>>> generator = pipeline(
... "text-generation",
... model="olm/olm-gpt2-latest",
... bad_words_ids=[[0,2]],
... )
>>> generator("")
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
[{'generated_text': ' Gasp '}]
```
The first token we get is `50256`:
```python
>>> generator.tokenizer.convert_ids_to_tokens(50256)
'ĠGasp'
```

You can fix this by modifying some parameters in the pipeline, but then you get a warning about the generation config:
```python
>>> generator = pipeline(
... "text-generation",
... model="olm/olm-gpt2-latest",
... bad_words_ids=[[0,2]],
... bos_token_id=0,
... pad_token_id=1,
... eos_token_id=2,
... )
>>> generator("")
/site-packages/transformers/generation/utils.py:1186: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
/site-packages/transformers/generation/utils.py:1273: UserWarning: Neither `max_length` nor `max_new_tokens` has been set, `max_length` will default to 50 (`generation_config.max_length`). Controlling `max_length` via the config is deprecated and `max_length` will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
warnings.warn(
[{'generated_text': ''}]
```

I have also posted this in the [olm/olm-gpt2-latest repo](https://huggingface.co/olm/olm-gpt2-latest/discussions/1#640a471a1ee054d66a6e1737).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.