eos_token_id for Textual Inversion

### Describe the bug

Hi, I implemented textual inversion follwowing this [link](https://huggingface.co/docs/diffusers/v0.32.2/en/training/text_inversion), but I think there is something wrong with `eos_token_id` in stable-diffusion-v1-5 text encoder [config](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/blob/main/text_encoder/config.json).

The config file is like this, this means `eos_token_id == 2`:
```
{
  | "_name_or_path": "openai/clip-vit-large-patch14",
  | "architectures": [
  | "CLIPTextModel"
  | ],
  | "attention_dropout": 0.0,
  | "bos_token_id": 0,
  | "dropout": 0.0,
  | "eos_token_id": 2,
  | "hidden_act": "quick_gelu",
  | "hidden_size": 768,
  | "initializer_factor": 1.0,
  | "initializer_range": 0.02,
  | "intermediate_size": 3072,
  | "layer_norm_eps": 1e-05,
  | "max_position_embeddings": 77,
  | "model_type": "clip_text_model",
  | "num_attention_heads": 12,
  | "num_hidden_layers": 12,
  | "pad_token_id": 1,
  | "projection_dim": 768,
  | "torch_dtype": "float32",
  | "transformers_version": "4.22.0.dev0",
  | "vocab_size": 49408
  | }
```

but in transformers modeling_clip.py,
```python
if self.eos_token_id == 2:
# The `eos_token_id` was incorrect before PR #24773: Let's keep what have been done here.
# A CLIP model with such `eos_token_id` in the config can't work correctly with extra new tokens added
# ------------------------------------------------------------
# text_embeds.shape = [batch_size, sequence_length, transformer.width]
# take features from the eot embedding (eot_token is the highest number in each sequence)
# casting to torch.int for onnx compatibility: argmax doesn't support int64 inputs with opset 14
pooled_output = last_hidden_state[
torch.arange(last_hidden_state.shape[0], device=last_hidden_state.device),
input_ids.to(dtype=torch.int, device=last_hidden_state.device).argmax(dim=-1),
]
```
I think this means current code is not compatible with textual inversion (because we just get the embedding of newly added token with token id 49408, not the eos token.

I might be wrong, but it will be really helpful for giving me any comments.

Thank you.

### Reproduction

accelerate launch textual_inversion.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--train_data_dir=$DATA_DIR \
--learnable_property="object" \
--placeholder_token="" \
--initializer_token="dog" \
--resolution=512 \
--train_batch_size=1 \
--gradient_accumulation_steps=4 \
--max_train_steps=3000 \
--learning_rate=5.0e-04 \
--scale_lr \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \

### System Info

diffusers 0.32.0.dev0

### Who can help?

_No response_