intel / intel/AI-Playground

Chat + image attachment fails: "URL scheme must be http, https, or data, got aipg-media:"

Open
#592 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
979
Forks
132
Avg merge
9h 23m
Merged PRs (30d)
8

Description

**Description:**

```markdown

In AI Playground **3.2.0-beta**, sending a **text + image** message in Chat mode fails immediately with:

```
URL scheme must be http, https, or data, got aipg-media:
```

followed by:

```
AI_NoOutputGeneratedError: No output generated. Check the stream for errors.
```

Text-only chat works fine. The failure only happens when an image is attached.

Root cause (from inspecting `app.asar`):

1. When an image is attached in Chat, the UI stores a file part using the custom media protocol:
- `{ type: "file", mediaType: "image/png", url: "aipg-media://media/...", filename: "..." }`

2. Before calling the model, there is conversion logic intended to turn `aipg-media://` URLs into `data:` URIs via `readAipgMediaAsBase64`. That logic still checks the **old** AI SDK shape:

```javascript
e.type === "file"
&& e.mediaType?.startsWith("image/")
&& typeof e.data === "string" // <-- false with current AI SDK
&& e.data.startsWith("aipg-media://")
```

3. The bundled AI SDK now normalizes file data into a **structured object**:

```javascript
// convertToModelMessages → mw(e.data)
{ type: "file", data: { type: "url", url: "aipg-media://media/..." }, mediaType: "image/png" }
```

Because `typeof e.data === "object"` (not `"string"`), the aipg-media → data-URI conversion is skipped.

4. AI SDK then validates the download URL (`validateDownloadUrl`) and rejects the custom scheme:

```text
URL scheme must be http, https, or data, got aipg-media:
```

So this looks like a format-mismatch regression between the chat image pipeline and the bundled AI SDK message model, not a local install/config problem.

## To Reproduce

Steps to reproduce the behavior:

1. Open AI Playground 3.2.0-beta on Windows.
2. Select Chat mode with a **vision-capable** local model (llama.cpp / OpenVINO backend).
3. Attach any local image (PNG/JPG) via the prompt-area `+` button or drag-and-drop.
4. Type any short prompt (e.g. "Describe this image").
5. Click Send.
6. See error toast/console:

```
[inference] inference/stream-failed: URL scheme must be http, https, or data, got aipg-media:
```

and

```
AI_NoOutputGeneratedError: No output generated. Check the stream for errors.
```

7. Remove the image and send text-only with the same model → succeeds.

## Expected behavior

Chat with an attached image should:

1. Resolve `aipg-media://` to a base64 `data:` URI (or equivalent inline bytes) **before** handing the message to the AI SDK / local backend.
2. Send the multimodal request successfully, same as text-only chat.

The existing conversion helper already knows how to do step 1 (`gE()` / `readAipgMediaAsBase64`); it just never runs because it still matches the old `data: string` shape.

### Suggested fix direction

Update the pre-request conversion (and the “has aipg-media image” detector) to also handle the current AI SDK shape, for example:

```javascript
// Current (misses structured data):
typeof e.data === "string" && e.data.startsWith("aipg-media://")

// Also match:
e.data?.type === "url" && typeof e.data.url === "string" && e.data.url.startsWith("aipg-media://")
// and/or
typeof e.url === "string" && e.url.startsWith("aipg-media://")
```

Then replace it with a data URI / `{ type: "data", data: base64 }` payload before `streamText` / chat transport.

## Screenshots

N/A (console error only). Full stack from renderer:

```
URL scheme must be http, https, or data, got aipg-media:
Error: URL scheme must be http, https, or data, got aipg-media:
at dist-BRzcC_7v.js:116:14657
at dist-BRzcC_7v.js:117:10246
at Array. (dist-BRzcC_7v.js:116:59168)
at kD.processQueue (dist-BRzcC_7v.js:116:59048)
...
at LD.sendMessage (dist-BRzcC_7v.js:117)
...
[unknown] global/unhandled-rejection: No output generated. Check the stream for errors.
AI_NoOutputGeneratedError: No output generated. Check the stream for errors.
```

## Environment (please complete the following information):

- OS: Windows 11
- GPU: Intel(R) Arc(TM) B580 Graphics (device id 0xE20B)
- CPU: Intel(R) Core(TM) i9-14900K (3.20 GHz)
- Version: AI Playground 3.2.0-beta
- Backend: llama.cpp chat backend (also seen with OpenVINO backend configured)
- Model: Qwen3.5-9B-Q4_K_M.gguf
- Install path: `E:\AI Playground`
- Product mode: studio
- Reproducibility: 100% when an image is attached; 0% for text-only

## Additional context

- Text-only chat with the same model/backend is stable.
- Unrelated noise in the same log session (not the cause of this bug):
- ComfyUI `insightface` wheel download from GitHub timed out (`uv.check-details.ComfyUI`).
- ComfyUI venv “environment doesn't match expected state” warning.
- The custom protocol is registered as privileged in the Electron main process (`registerSchemesAsPrivileged([{ scheme: "aipg-media", ... }])`), so the UI can preview images locally; the chat request path just never rewrites that URL for the model.
- This blocks any multimodal chat use case on 3.2.0-beta and makes image understanding unusable until fixed.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Chat image-message conversion and aipg-media detector in the source corresponding to the bundled app.asar; inspect readAipgMediaAsBase64 and the data shape passed before streamText or the chat transport. Reproduce with a local image and a vision-capable model, then verify that the structured URL is converted before AI SDK validation and that text-only chat remains working.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.