[BUG] Galactica-6.7B: Asking to pad but the tokenizer does not have a padding token

Describe the bug

Galactica model running text-generation does not have a pad token.

git clone https://github.com/microsoft/DeepSpeedExamples.git
cd DeepSpeedExamples/inference/huggingface/text-generation
deepspeed --num_gpus 8 inference-test.py --name facebook/galactica-6.7b --ds_inference --use_meta_tensor --use_kernel

Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "inference-test.py", line 135, in <module>
    outputs = pipe(inputs,
  File "/data0/pratyush/DeepSpeedExamples/inference/huggingface/text-generation/utils.py", line 69, in __call__
    outputs = self.generate_outputs(input_list, num_tokens=num_tokens, do_sample=do_sample)    
  File "/data0/pratyush/DeepSpeedExamples/inference/huggingface/text-generation/utils.py", line 107, in generate_outputs
    input_tokens = self.tokenizer.batch_encode_plus(inputs, return_tensors="pt", padding=True) 
  File "/data0/pratyush/miniconda3/envs/ds/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2829, in batch_encode_plus
    padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
  File "/data0/pratyush/miniconda3/envs/ds/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2466, in _get_padding_truncation_strategies
    raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.

I tried to set a pad token / replace the tokenizer in the following three ways:

1. pipe.tokenizer.pad_token = pipe.tokenizer.eos_token

2. pipe.tokenizer.add_special_tokens({'pad_token': '[PAD]'})

3. pipe.tokenizer = AutoTokenizer.from_pretrained("facebook/galactica-6.7b")

Yet, 1 and 3 still result in the same error as above. 2 yields:

ValueError: The following `model_kwargs` are not used by the model: ['token_type_ids'] (note: typos in the generate arguments will also show up in this list)

Expected behavior
Updated tokenizer should work, at least?

A couple of other related questions:

1. Is there a list of supported models for DeepSpeed Inference w/ meta tensors?
2. I read that DeepSpeed-Inference supports automatic tensor parallelism for Huggingface models by default, but I need meta_tensors to load larger models. However, meta_tensors seem to require kernel injection (at least for OPT models, which error out otherwise), so I've been using that. Is my understanding correct / is that the right approach?

Rest of the setup is exactly the same as in https://github.com/microsoft/DeepSpeed/issues/3871

Thanks!

Reproduce the command in inference/huggingface/text-generation/inference-test.py, then read the tokenizer and generation flow in inference/huggingface/text-generation/utils.py, especially the calls around lines 69 and 107. Check how padding and model inputs are passed through batch_encode_plus and generation; done means Galactica text generation runs with the updated tokenizer without the padding or unused model_kwargs errors.