missing pad_token error when using GPT2Chinese
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 487
- PR merge metrics
- No merged PRs in 30d
Description
### π Describe the bug
When i run the code in summarization-rlhf using GPT2Chinese, the following error occurs . I have checked the "specail_tokens_map.json", it does have the "[PAD]" token. Any and all help is appreciated.
Errno 2] No such file or directory: '/tmp/ipykernel_31/1112272871.py' β
β β
β /kaggle/working/trlx/trlx/trlx.py:105 in train β
β β
β 102 β β if eval_prompts is None: β
β 103 β β β eval_prompts = prompts[:batch_size] β
β 104 β β β
β β± 105 β β trainer.make_experience(config.method.num_rollouts) β
β 106 β β
β 107 β # Offline training from the collected samples (e.g. SFT, ILQL) β
β 108 β elif samples: β
β β
β /kaggle/working/trlx/trlx/trainer/accelerate_ppo_trainer.py:281 in make_experience β
β β
β 278 β β while len(ppo_rl_elements) < num_rollouts: β
β 279 β β β stats = {} β
β 280 β β β # Get next batch in prompt dataset β
β β± 281 β β β batch: PromptBatch = next(self.prompt_iterator) β
β 282 β β β β
β 283 β β β rollout_generate_time = time() β
β 284 β
β β
β /kaggle/working/trlx/trlx/utils/__init__.py:240 in infinite_dataloader β
β β
β 237 β Returns a cyclic infinite dataloader from a finite dataloader β
β 238 β """ β
β 239 β for _ in repeat(dataloader): β
β β± 240 β β yield from dataloader β
β 241 β
β β
β /opt/conda/lib/python3.10/site-packages/accelerate/data_loader.py:378 in __iter__ β
β β
β 375 β β dataloader_iter = super().__iter__() β
β 376 β β # We iterate one batch ahead to check when we are at the end β
β 377 β β try: β
β β± 378 β β β current_batch = next(dataloader_iter) β
β 379 β β except StopIteration: β
β 380 β β β yield β
β 381 β
β β
β /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:634 in __next__ β
β β
β 631 β β β if self._sampler_iter is None: β
β 632 β β β β # TODO(https://github.com/pytorch/pytorch/issues/76750) β
β 633 β β β β self._reset() # type: ignore[call-arg] β
β β± 634 β β β data = self._next_data() β
β 635 β β β self._num_yielded += 1 β
β 636 β β β if self._dataset_kind == _DatasetKind.Iterable and \ β
β 637 β β β β β self._IterableDataset_len_called is not None and \ β
β β
β /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:678 in _next_data β
β β
β 675 β β
β 676 β def _next_data(self): β
β 677 β β index = self._next_index() # may raise StopIteration β
β β± 678 β β data = self._dataset_fetcher.fetch(index) # may raise StopIteration β
β 679 β β if self._pin_memory: β
β 680 β β β data = _utils.pin_memory.pin_memory(data, self._pin_memory_device) β
β 681 β β return data β
β β
β /opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:54 in fetch β
β β
β 51 β β β β data = [self.dataset[idx] for idx in possibly_batched_index] β
β 52 β β else: β
β 53 β β β data = self.dataset[possibly_batched_index] β
β β± 54 β β return self.collate_fn(data) β
β 55 β
β β
β /kaggle/working/trlx/trlx/pipeline/offline_pipeline.py:160 in collate_fn β
β β
β 157 β β
β 158 β def create_loader(self, batch_size: int, shuffle=False) -> DataLoader: β
β 159 β β def collate_fn(xs): β
β β± 160 β β β out = self.tokenizer.pad([{"input_ids": x["input_ids"]} for x in xs], return β
β 161 β β β β
β 162 β β β for key in xs[0]: β
β 163 β β β β if key != "input_ids" and key != "attention_mask": β
β β
β /opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2995 in pad β
β β
β 2992 β β β β encoded_inputs[key] = to_py_obj(value) β
β 2993 β β β
β 2994 β β # Convert padding_strategy in PaddingStrategy β
β β± 2995 β β padding_strategy, _, max_length, _ = self._get_padding_truncation_strategies( β
β 2996 β β β padding=padding, max_length=max_length, verbose=verbose β
β 2997 β β ) β
β 2998 β
β β
β /opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2443 in β
β _get_padding_truncation_strategies β
β β
β 2440 β β β
β 2441 β β # Test if we have a padding token β
β 2442 β β if padding_strategy != PaddingStrategy.DO_NOT_PAD and (not self.pad_token or sel β
β β± 2443 β β β raise ValueError( β
β 2444 β β β β "Asking to pad but the tokenizer does not have a padding token. " β
β 2445 β β β β "Please select a token to use as `pad_token` `(tokenizer.pad_token = tok β
β 2446 β β β β "or add a new pad token via `tokenizer.add_special_tokens({'pad_token': β
β°βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ―
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as
`pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via
`tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
### Which trlX version are you using?
_No response_
### Additional system and package information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.