vertexai.rag.upload_file ignores credentials from vertexai.init and fails with invalid_scope

#### Environment details

- OS type and version: macOS 15.6.1 (Apple Silicon, arm64)
- Python version: `3.13.8`
- pip version: `pip 25.2`
- google-cloud-aiplatform: `1.122.0`

Service account is provided via `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json`
The same service account works fine for Vertex / Gemini calls.

---

#### Steps to reproduce

1. Set `GOOGLE_APPLICATION_CREDENTIALS` to a valid service account JSON.
2. Create a small script like this:

```python
import os
from google.oauth2 import service_account
from google.auth.transport.requests import Request
import vertexai
from vertexai import rag

PROJECT_ID = "my-project-id"
LOCATION = "my-region" # region where RAG Engine is enabled
LOCAL_FILE = "my-local-test-file.pdf"

def build_sa_creds():
sa_path = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
creds = service_account.Credentials.from_service_account_file(sa_path, scopes=scopes)
creds.refresh(Request()) # ✅ works
return creds

def main():
# These creds work for Gemini / Vertex calls
creds = build_sa_creds()

vertexai.init(project=PROJECT_ID, location=LOCATION, credentials=creds)

# This call succeeds:
corpus = rag.create_corpus(display_name="test-corpus")
print("Created corpus:", corpus.name)

# This call fails:
rag_file = rag.upload_file(
corpus_name=corpus.name,
path=LOCAL_FILE,
display_name="my-local-test-file.pdf",
description="Repro for invalid_scope in upload_file",
)

if __name__ == "__main__":
main()

```

#### Code example

(See above for full script.)

We also tested a monkeypatch to confirm the root cause:

```python
from unittest.mock import patch

def fake_auth_default(scopes=None, quota_project_id=None, request=None):
# Reuse the same SA creds with explicit cloud-platform scope
return build_sa_creds(), PROJECT_ID

with patch("vertexai.rag.rag_data.auth.default", new=fake_auth_default):
vertexai.init(project=PROJECT_ID, location=LOCATION, credentials=build_sa_creds())
corpus = rag.create_corpus(display_name="patched-corpus")
rag_file = rag.upload_file(
corpus_name=corpus.name,
path=LOCAL_FILE,
display_name="test-file.pdf",
)
print("✅ upload_file succeeded with patched auth.default:", rag_file.name)
```

With this patch, upload_file works correctly.

Stack trace

Without the monkeypatch, rag.upload_file fails with:

File ".../site-packages/vertexai/rag/rag_data.py", line 441, in upload_file
raise RuntimeError("Failed in uploading the RagFile due to: ", e) from e
RuntimeError: ('Failed in uploading the RagFile due to: ',
RefreshError('invalid_scope: Invalid OAuth scope or ID token audience provided.', {...}))

#### Additional details / suspected cause

Looking at the implementation in vertexai/rag/rag_data.py, upload_file currently does:

```python

credentials, _ = auth.default()
authorized_session = google_auth_requests.AuthorizedSession(credentials=credentials)
response = authorized_session.post(...)

```

This means:

- upload_file ignores any credentials passed via vertexai.init(project=..., location=..., credentials=...).

- It always uses google.auth.default() internally, and whatever default scope / audience that returns in the current environment.

In our project:

- If we explicitly build SA credentials with cloud-platform scope (as shown above), they work for all other Vertex/Gemini operations.

- But the credentials implicitly returned by auth.default() (inside upload_file) fail with invalid_scope when calling the RAG upload endpoint.

When we monkeypatch vertexai.rag.rag_data.auth.default to return our explicitly-scoped SA credentials, rag.upload_file starts working reliably.

Expected behaviour

Ideally, upload_file would:

- Either reuse the credentials passed to vertexai.init(...), or

- Allow explicit credentials to be passed in, or

- At least document that it always uses google.auth.default() and may ignore the init credentials.

At the moment it’s a bit surprising that:

- rag.create_corpus works with our credentials,

- Gemini model calls work,

- but rag.upload_file fails unless we patch auth.default.

Thanks!