Function Call doesn't seem to work with images as part of the prompts

When configuring Gemini/Vertex AI with a function call and a prompt that includes an image, VertexAI/Gemini throws a 500 error with no description of what the issue actually is.

#### Environment details

- Programming language: TS
- OS: Linux
- Language runtime version: Node JS v20.x
- Package version: 1.2.0
- Gemini 1.5 Pro Preview 0514

#### Steps to reproduce

1. Create a function call whilst passing in an image as part of the prompt
2. Gemini/Vertex AI throws 500 error

In the documentation, all the examples for doing function calls are with text prompts, so I may just be doing something that is not supported, but I also couldn't find anything in the docs that said images are NOT supported as part of prompts for function calls.

Additionally, I have tested my prompt without the image as part of the context, and it does the function call as you would expect.

Please NOTE: I can reproduce this with both the vertex ai library and a straight up curl call as well, so this is most likely a Gemini issue rather than a library issue, but since I don't have a google support contract, I can't really open a support ticket, so for the sake of trying to bring visibility to this issue, I'm filing it here. Apologies if there is a better venue available.

POST API endpoint:
```
https://us-central1-aiplatform.googleapis.com/v1/projects//locations/us-central1/publishers/google/models/gemini-1.5-pro-preview-0514:generateContent
```

What is being passed to Vertex AI:
```json
{
"contents": [
{
"role": "user",
"parts": [
{
"inlineData": {
"mimeType": "image/jpeg",
"data": "base64ofimage"
}
},
{
"text": "categorize the image"
}
]
}
],
"tools": [
{
"functionDeclarations": [
{
"name": "categorize",
"description": "accepts the categorized guess from model and stores it in API cache",
"parameters": {
"type": "object",
"properties": {
"category": { "type": "string", "description": "the category of the image" }
}
}
}
]
}
],
"generationConfig": {
"temperature": 1,
"topP": 0.95
},
"safetySettings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
}
]
}
```

response:
```json
{
"error": {
"code": 500,
"message": "Internal error encountered.",
"status": "INTERNAL"
}
}
```

a working JSON with only a text prompt:
```json
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "categorize an old chandelier"
}
]
}
],
"tools": [
{
"functionDeclarations": [
{
"name": "categorize",
"description": "accepts the categorized guess from model and stores it in API cache",
"parameters": {
"type": "object",
"properties": {
"category": { "type": "string", "description": "the category of the object, example boats" }
}
}
}
]
}
],
"generationConfig": {
"temperature": 1,
"topP": 0.95
},
"safetySettings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"threshold": "BLOCK_MEDIUM_AND_ABOVE"
}
]
}
```

Inizia riproducendo la richiesta sull’endpoint documentato Vertex AI generateContent, confrontando il payload con immagine e function-call con il payload di solo testo. Determina se l’errore si verifica nella libreria TypeScript o nel servizio Gemini upstream; done deve includere un comportamento supportato confermato, un errore utile o un aggiornamento della documentazione, oppure prove che il problema appartenga all’upstream.

Scritto dal modello di indicizzazione a partire dal testo della issue.