Prediction API errors out with retriable errors

#### Environment details

- OS type and version: gLinux
- Python version: 3.8.9
- pip version: 22.0.4
- `google-cloud-aiplatform version: 1.25.0

#### Steps to reproduce

Get document embeddings using langchain, and upon a 503 ServiceUnavailable error see it be thrown out:

#### Code example

I am working with the question-answering example from Langchain but using VertexAI embeddings.

```python
loader = DirectoryLoader("./ingested/")
raw_documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
documents = text_splitter.split_documents(raw_documents)
embeddings = VertexAIEmbeddings()
print('Calling vector store...')
vectorstore = FAISS.from_documents(documents, embeddings, batch_size=5)
```

#### Stack trace

```
Traceback (most recent call last):
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "recvmsg:Connection reset by peer"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-05-31T19:41:22.051480349-07:00", grpc_status:14, grpc_message:"recvmsg:Connection reset by peer"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "ingest.py", line 29, in
ingest_docs()
File "ingest.py", line 21, in ingest_docs
vectorstore = FAISS.from_documents(documents, embeddings, batch_size=5)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/langchain/vectorstores/base.py", line 307, in from_documents
return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/langchain/vectorstores/faiss.py", line 425, in from_texts
embeddings = embedding.embed_documents(texts)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/langchain/embeddings/vertexai.py", line 41, in embed_documents
embeddings_batch = self.client.get_embeddings(text_batch)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/vertexai/language_models/_language_models.py", line 498, in get_embeddings
prediction_response = self._endpoint.predict(
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/google/cloud/aiplatform/models.py", line 1546, in predict
prediction_response = self._prediction_client.predict(
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 602, in predict
response = rpc(
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
return wrapped_func(*args, **kwargs)
File "/home/pabloem/codes/chat-langchain/venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 recvmsg:Connection reset by peer

```

Start at google/cloud/aiplatform/models.py in predict, then follow the call into prediction_service/client.py and google/api_core/grpc_helpers.py shown in the traceback. Reproduce the 503 with the VertexAI embeddings example and determine the expected handling for retriable ServiceUnavailable errors; done means the behavior is covered by an appropriate regression test.

由索引模型根据 Issue 内容生成。