MemberJunction / MemberJunction/MJ

GoogleDriveFileStorage: initialize() discards a working service-account driver, and PutObject sends a Buffer where googleapis needs a stream

Open
#3,847 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
308

Description

Two independent defects in `@memberjunction/storage`'s Google Drive driver (seen on `6.1.0-edge.1`, googleapis `171.4.0`). Together they make a **service-account-backed Drive account fail with no usable diagnosis**, and the second one fails *after* auth and folder creation succeed — so every symptom reads as a storage-configuration problem.

Found while tracing why consented realtime recordings never reached storage: `AIAgentSession` rows carried `RecordingMedia='Audio'` and a stamped `RecordingStartedAt`, but `RecordingFileID` was null and `__mj.File` was empty.

---

## 1. `initialize(config)` throws away the client the constructor already built

`GoogleDriveFileStorage`'s **constructor** supports three auth methods — key file (service account), credentials JSON (service account), and OAuth2 refresh token.

`initialize(config)` — the path taken when credentials come from a **database** `MJ: Credentials` record via `initializeDriverWithAccountCredentials` — reads only `clientID` / `clientSecret` / `refreshToken`:

```js
if (this._clientID && this._clientSecret && this._refreshToken) { /* OAuth2 */ }
else {
throw new Error('Google Drive storage requires clientID, clientSecret, and refreshToken to be set');
}
```

So:

* a **service account** supplied through the DB credential is ignored and the driver throws;
* worse, it throws **even when `this._drive` was already successfully built by the constructor** from `STORAGE_GDRIVE_KEY_FILE` / `STORAGE_GDRIVE_CREDENTIALS_JSON`. Attaching a credential record *breaks a driver that was working from env*.

**Suggested fix:** mirror the constructor's methods in `initialize` (accept `keyFile` / `credentialsJSON`), and do not throw when a client is already initialized.

## 2. `PutObject` passes a Buffer where googleapis requires a stream

Past #1, the driver authenticates, navigates and creates the folder tree correctly, then dies inside `files.create`:

```
TypeError: part.body.pipe is not a function
at Resource$Files.create (googleapis/build/src/apis/drive/v3.js:943:65)
at GoogleDriveFileStorage.PutObject (GoogleDriveFileStorage.js:1067:41)
```

`media.body` is passed the raw `data` Buffer; googleapis 171.x assembles the multipart request by calling `.pipe()` on it. Both the `files.create` and `files.update` branches are affected.

**Suggested fix:** `body: Buffer.isBuffer(data) ? Readable.from(data) : data`.

This one is the expensive defect, because it surfaces **after** auth and folder creation have succeeded — the failure looks like bad storage config rather than a client-library incompatibility.

## 3. Related gap: Shared Drives are unreachable

Past both fixes, Drive answers honestly:

```
403 — Service Accounts do not have storage quota.
Leverage shared drives, or use OAuth delegation instead.
```

That is a Google constraint, not an MJ bug — but the driver never passes `supportsAllDrives` / `includeItemsFromAllDrives` / `driveId`, so a Shared Drive **cannot be targeted even when one is configured**. As it stands, service-account auth cannot work with this driver at all. Worth treating as part of the same fix.

---

## Reproduction

Any `MJ: File Storage Accounts` row on the Google Drive provider whose credential carries service-account material rather than OAuth2 fields, then any `UploadFile` against it. Nothing needs to be realtime-specific.

## Note on diagnosability

Every layer above this swallows: `storeRealtimeRecording` logs and returns null, the resolver returns `{Success:false}` without throwing, and `__mj.ErrorLog` stays empty. That posture is right — a storage failure must not fail a session — but with the driver error discarded too, there is no durable trace of *why* an upload failed. Surfacing the driver's error on the failure result would have turned this investigation into one log line.

Contributor guide

Open the contributing guide

Research direction

Start in GoogleDriveFileStorage.js, focusing on initialize() and the PutObject files.create/files.update branches. Reproduce the issue with a Google Drive storage account using service-account credentials and UploadFile, then verify that existing clients remain usable, uploads complete after authentication and folder creation, and configured Shared Drive behavior is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, javascript
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.