PipedreamHQ / PipedreamHQ/pipedream
[FEATURE] google_vertex_ai: accept inline/base64 file input in Analyze Image/Video (currently GCS-only)
- Dominant language
- JavaScript
- Stars
- 11.7k
- Forks
- 5.8k
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 102
Description
**Is your feature request related to a problem? Please describe.**
`google_vertex_ai-analyze-image-video` only accepts a GCS URI. Its `url` prop says "Only GCS URIs are supported", and `run()` hardcodes the Gemini `fileData`/`fileUri` part shape:
```js
parts: [
{ fileData: { mimeType: this.mimeType, fileUri: this.url } },
{ text: this.instructions },
]
```
So analyzing an image that isn't already in Cloud Storage means standing up a bucket, granting the connected account write access, and uploading the file first — just to read it back. That's a hard blocker for the common case of an image captured or fetched at workflow runtime (a page screenshot, an attachment, an HTTP response body), where the bytes are already in hand and no bucket exists.
`generateContent` itself has no such restriction: `Part` accepts `inlineData` (a `Blob` of `mimeType` + base64 `data`) alongside `fileData`. The limitation is in the action, not the API.
**Describe the solution you'd like**
Accept inline file bytes in `analyze-image-video` and send `inlineData` instead of `fileData` when the input isn't a `gs://` URI.
The app already has this pattern — `generate-video-from-image` in the same package takes "either a file URL or a path to a file in the `/tmp` directory", runs it through `getFileStreamAndMetadata()` from `@pipedream/platform`, converts with its own `streamToBase64()` helper, and sends `bytesBase64Encoded` plus the `mimeType` from the stream metadata. Reusing that in `analyze-image-video` would make the two actions consistent and needs no new dependency.
Ideally the prop accepts all of: a `gs://` URI (current behavior, mapped to `fileData`), a file URL or `/tmp` path (mapped to `inlineData`), and a raw base64 string for callers that already hold the bytes. Deriving `mimeType` from the stream metadata when available would also drop the manual `mimeType` prop for most users — it currently defaults to `image/jpeg` regardless of the actual file.
The same applies to the app's `generateContent()` method if it's ever exposed for other multimodal actions.
**Do you have a workaround?**
Only by bypassing the action. We call the Vertex `:generateContent` endpoint directly through the Connect API proxy, reusing the same connected `google_vertex_ai` account, and build the `inlineData` part ourselves. That works and confirms the credential and scopes are sufficient — the packaged action is the only thing that can't express the request.
The other workaround, provisioning a GCS bucket purely as a staging area for bytes we already have, adds storage setup, IAM, lifecycle cleanup, and a round trip for a single synchronous call.
**Comparable features in other tools?**
The clearest comparison is inside this app: `generate-video-from-image` already accepts a URL or `/tmp` path and base64-encodes it. `analyze-image-video` is the inconsistent one.
More broadly, the Google Gen AI SDKs and the REST API both treat inline bytes as a first-class input to `generateContent`, and GCS as the option for files too large to inline.
**Additional context**
- App: `google_vertex_ai` (`components/google_vertex_ai`, `@pipedream/google_vertex_ai@0.2.0`)
- Action: `components/google_vertex_ai/actions/analyze-image-video/analyze-image-video.mjs` (v0.0.3)
- Prior art in the same package: `components/google_vertex_ai/actions/generate-video-from-image/generate-video-from-image.mjs` (v0.0.2) and `actions/common/generate-video.mjs` (`streamToBase64`)
- `Part` / `Blob` reference: https://cloud.google.com/vertex-ai/docs/reference/rest/v1/Content
- `generateContent` reference: https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.publishers.models/generateContent
Unrelated to the input shape, but affecting the same action: it still pins `model: "gemini-1.0-pro-vision"`, which has been retired. Worth refreshing in the same pass, or the action fails before the input format matters.
Contributor guide
Assessment
This issue has not been assessed yet.