Inconsistent Probability Scores Between GCS URI and Image Bytes Input

#### Environment details

- Programming language: Python
- OS: MacOS
- Language runtime version: Python 3.12.9
- Package version: 1.67.0

#### Steps to reproduce

# Bug Report: Inconsistent Probability Scores Between GCS URI and Image Bytes Input for fine tuned model (Flash 2.5 Lite)

## Summary

The Gemini API (via `google.genai` library) returns **significantly different probability scores** (avg_logprobs) for identical images depending on whether the image is provided as a GCS URI or as raw image bytes, even when using identical configuration parameters.

## Environment

- **Library**: `google-genai` with Vertex AI
- **Model**: Fine-tuned Gemini model on Vertex AI
- **Client**: `genai.Client(vertexai=True)`
- **Project**: `hidden`
- **Location**: `us-east4`
- **API Version**: `v1`
- **Language**: Python

## Reproduction Steps

### Setup Code

```python
from google import genai
from google.genai.types import HttpOptions
from google.genai import types
from google.cloud import storage
import math

# Initialize clients
client = genai.Client(
vertexai=True,
project="account-name",
location="us-east4",
http_options=HttpOptions(api_version="v1")
)
storage_client = storage.Client(project="account-name")

# Configuration (identical for both methods)
config = {
"temperature": 0,
"top_p": 1.0,
"top_k": 1,
"candidate_count": 1,
"response_logprobs": True,
"seed": 42,
}

model_name = "projects/{account_id}/locations/us-east4/endpoints/{endpoint}"
instructions = "Your instruction text here"
```

### Method 1: GCS URI (Direct Reference)

```python
gcs_uri = "gs://{bucket}/{path-to-image}.jpg"

response1 = client.models.generate_content(
model=model_name,
contents=[gcs_uri, instructions],
config=config
)

prediction1 = response1.text.strip()
probability1 = math.exp(response1.candidates[0].avg_logprobs)
```

### Method 2: Image Bytes (Downloaded First)

```python
# Parse and download from GCS
uri_parts = gcs_uri[5:].split("/", 1)
bucket_name = uri_parts[0]
blob_path = uri_parts[1]

bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_path)
image_data = blob.download_as_bytes()

# Create image part from bytes
image = types.Part.from_bytes(data=image_data, mime_type="image/jpeg")

response2 = client.models.generate_content(
model=model_name,
contents=[image, instructions],
config=config
)

prediction2 = response2.text.strip()
probability2 = math.exp(response2.candidates[0].avg_logprobs)
```

## Expected Behavior

Both methods should return **identical** or **nearly identical** probability scores since:
- ✅ Same image content (byte-for-byte identical - **SHA-256 hash verified**)
- ✅ Same model
- ✅ Same configuration parameters
- ✅ Same instructions
- ✅ Deterministic settings (`temperature=0`, `seed=42`)

**Image Identity Verification:**
```python
import hashlib

# Download image and compute hash
image_bytes = blob.download_as_bytes()
sha256_hash = hashlib.sha256(image_bytes).hexdigest()
# Hash: [verified in test notebook]
# File size: [verified in test notebook]
```

Both methods use the exact same GCS object - the downloaded bytes in Method 2 are **proven identical** to the image referenced by the URI in Method 1.

## Actual Behavior

**Dramatically different probability scores:**

| Method | Input Type | Prediction | Probability | Difference |
|--------|-----------|------------|-------------|------------|
| GCS URI | `gs://bucket/path/image.jpg` | `0` | **0.8365** | - |
| Image Bytes | `types.Part.from_bytes()` | `0` | **0.9999** | **+0.163** (19.5% increase) |

### Additional Testing

We verified this finding by ensuring both methods use **identical configuration parameters**:

```python
# Same config for both methods
config = {
"temperature": 0,
"top_p": 1.0,
"top_k": 1,
"candidate_count": 1,
"response_logprobs": True,
"seed": 42,
}
```

| Method | Input Type | Config | Probability |
|--------|-----------|--------|-------------|
| 1 | GCS URI | Identical | **0.8365** |
| 2 | Image Bytes | Identical | **0.9999** |

**Key Finding**: The probability difference persists even with identical configuration parameters. The difference is **entirely** due to the image input format.

## Impact

### 1. **Inconsistent Confidence Scores**
Production systems using different input methods will report vastly different confidence levels for identical predictions. A 20% probability difference is significant.

### 2. **Non-Deterministic Behavior**
Despite setting `temperature=0` and `seed=42`, the API produces different probability scores based solely on input format.

### 3. **API Contract Violation**
The same configuration parameters should produce deterministic results regardless of whether the image is provided as a URI or bytes.

### 4. **Evaluation/Production Mismatch**
- Models evaluated using GCS URIs (common in batch processing) show different probability distributions than production systems using image bytes
- This breaks the fundamental assumption that evaluation metrics reflect production performance

### 5. **Real-World Consequences**
- Confidence-based thresholds behave differently in evaluation vs. production
- Monitoring systems may incorrectly flag drift or anomalies
- Decision-making systems relying on probability scores may behave unpredictably

## Hypothesis

The Gemini API likely uses different internal code paths for processing:
- **GCS URIs**: May apply additional preprocessing, caching, or use a different inference pipeline
- **Image Bytes**: Direct inference path with different probability calculation

This results in different `avg_logprobs` values even with identical model, images, and configuration.

## Additional Notes

- ✅ **Image content verified identical**: SHA-256 hash confirms both methods process the exact same bytes
- ✅ **Predictions remain identical** (both output `0`) - only probability scores differ
- ✅ **Reproducible** across multiple test images
- ✅ **Consistent pattern**: GCS URIs always produce lower probability scores (~0.84), bytes always produce higher scores (~0.99)
- ❌ Configuration parameters appear to be **ignored or handled differently** when using GCS URIs

## Workaround

For consistent probability scores, always use the same input method:
- **Option A**: Always download images and send bytes
- **Option B**: Always use GCS URIs (but accept that config params may not work as expected)

We have opted for Option A to ensure configuration parameters are respected.

## Expected Fix

The API should:
1. Return identical (or nearly identical) probability scores for identical inputs regardless of format
2. Respect all configuration parameters (`top_p`, `top_k`, `candidate_count`) consistently for both input methods
3. Document any known differences in behavior between GCS URI and bytes input methods