Sharing MoE results for integration tests

Hi there! 👋

We have been working to add Switch transformers in Hugging Face `transformers` in https://github.com/huggingface/transformers/pull/19323 with @ArthurZucker!
We believe that we have almost finalized the conversion process, and would like to ask you few questions to make sure the conversion has been done correctly.
We are currently matching the logits between the output that is obtained with `t5x.infer` and our forward pass, given the same input. Therefore we have tried to play a bit with the models and do some generations. The generations can be done in a T5-style using `transformers` as follows:
```
from transformers import SwitchTransformersForConditionalGeneration, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = SwitchTransformersForConditionalGeneration.from_pretrained("switch-base-8")

input_text = "summarize: Hello world"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
output = model.generate(input_ids)

print(tokenizer.decoder(output[0]))
>>> The best way to do it is to use a smartphone. Hello there
```
The model seems to output gibberish for T5 tasks such as translation, but seems to produce intelligible output for summarization (even though it is not great). Are the weights that has been open-sourced for the Switch transformers pre-trained weights (ie not fine-tuned)? Therefore not "ready-to-use" as Flan-T5?

We are observing the same behavior for `t5_v1_1` models that have been open-sourced and converted, if my understanding is correct, Flan-T5 is the only model that is "ready-to-use" after downloading the weights, other models needs to be fine-tuned - therefore the output that we are currently getting with our converted MoE seems to be consitent/correct.

Also it seems that our converted model perform decent results on MLM tasks, see for example:

```
input_ids = tokenizer("A walks into a bar a orders a .", return_tensors="pt").input_ids.to(torch_device)
sequences = model.generate(input_ids)
output_str = tokenizer.batch_decode(sequences, skip_special_tokens=False)[0]
>>> man drink.
```

Let us know if we are missing something! Would love to discuss further details and introduce together the first MoE model in `transformers`
Thanks!

Start by reviewing the referenced transformers PR and the t5x.infer and SwitchTransformersForConditionalGeneration entry points, then reproduce the model.generate examples in the issue. Done would require an agreed answer on whether the converted weights and generated outputs are correct, plus a clear integration task; no repository file or test is identified here.

Written by the indexing model from the issue text.