Function Call doesn't seem to work with images as part of the prompts

When configuring Gemini/Vertex AI with a function call and a prompt that includes an image, VertexAI/Gemini throws a 500 error with no description of what the issue actually is.

#### Environment details

- Programming language: TS
- OS: Linux
- Language runtime version: Node JS v20.x
- Package version: 1.2.0
- Gemini 1.5 Pro Preview 0514

#### Steps to reproduce

1. Create a function call whilst passing in an image as part of the prompt
2. Gemini/Vertex AI throws 500 error

In the documentation, all the examples for doing function calls are with text prompts, so I may just be doing something that is not supported, but I also couldn't find anything in the docs that said images are NOT supported as part of prompts for function calls.

Additionally, I have tested my prompt without the image as part of the context, and it does the function call as you would expect.

Please NOTE: I can reproduce this with both the vertex ai library and a straight up curl call as well, so this is most likely a Gemini issue rather than a library issue, but since I don't have a google support contract, I can't really open a support ticket, so for the sake of trying to bring visibility to this issue, I'm filing it here. Apologies if there is a better venue available.

POST API endpoint:
```
https://us-central1-aiplatform.googleapis.com/v1/projects//locations/us-central1/publishers/google/models/gemini-1.5-pro-preview-0514:generateContent
```

What is being passed to Vertex AI:
```json
{
"contents": [
{
"role": "user",
"parts": [
{
"inlineData": {
"mimeType": "image/jpeg",
"data": "base64ofimage"
}
},
{
"text": "categorize the image"
}
]
}
],
"tools": [
{
"functionDeclarations": [
{
"name": "categorize",
"description": "accepts the categorized guess from model and stores it in API cache",
"parameters": {
"type": "object",
"properties": {
"category": { "type": "string", "description": "the category of the image" }
}
}
}
]
}
],
"generationConfig": {
"temperature": 1,
"topP": 0.95
},
"safetySettings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
}
]
}
```

response:
```json
{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
```

a working JSON with only a text prompt:
```json
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "categorize an old chandelier"
}
]
}
],
"tools": [
{
"functionDeclarations": [
{
"name": "categorize",
"description": "accepts the categorized guess from model and stores it in API cache",
"parameters": {
"type": "object",
"properties": {
"category": { "type": "string", "description": "the category of the object, example boats" }
}
}
}
]
}
],
"generationConfig": {
"temperature": 1,
"topP": 0.95
},
"safetySettings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
}
]
}
```

Beginne damit, die Anfrage gegen den dokumentierten Vertex AI generateContent-Endpunkt zu reproduzieren, und vergleiche dabei die Payload mit Bild plus function-call mit der reinen Text-Payload. Bestimme, ob der Fehler in der TypeScript-Bibliothek oder im vorgeschalteten Gemini-Dienst auftritt; als erledigt gilt die Aufgabe, wenn entweder ein bestätigtes unterstütztes Verhalten, ein nützlicher Fehler oder eine Dokumentationsaktualisierung vorliegt oder wenn Belege dafür vorliegen, dass das Problem beim Upstream liegt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.