yetone / yetone/alma-plugins

Plugin API limitation: chat.message.willSend only exposes content, not multipart image parts

Open
#23 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
53
Forks
17
Avg merge
2h 5m
Merged PRs (30d)
1

Description

## Summary

`chat.message.willSend` is currently documented as a text-only hook (`input.content` / `output.content`), but real user messages with uploaded images are stored as multipart `message.parts[]` with `type: "file"` image entries.

This makes it effectively impossible to build a transparent image-aware transform plugin (for example, a vision proxy that auto-describes images for non-vision models) using the current public plugin API.

## Why this matters

I was trying to build an Alma plugin similar to `pi-vision-proxy`:

- user sends text + image
- plugin intercepts before send
- plugin routes the image to a vision-capable model
- plugin injects the generated description back into the outgoing message
- non-vision models can then “see” the image through the injected description

This workflow depends on being able to inspect and modify image-bearing outgoing messages.

## What the docs currently say

The docs for `chat.message.willSend` define the hook as:

```ts
interface ChatMessageWillSendInput {
threadId: string;
content: string;
model: string;
providerId: string;
}

interface ChatMessageWillSendOutput {
content?: string;
cancel?: boolean;
}
```

So the documented mutation surface is text-only.

## What real thread data looks like

However, when a user uploads an image, the actual stored message shape is multipart and uses `parts`:

```json
{
"message": {
"role": "user",
"parts": [
{
"type": "text",
"text": "这是什么照片"
},
{
"type": "file",
"mediaType": "image/jpeg",
"url": "http://localhost:23001/api/gallery/cache/upload-...jpg",
"filename": "sport_02_with_id.jpg"
}
]
}
}
```

So there is a mismatch:

- real message structure: `message.parts[]`
- public hook mutation surface: `content` only

## Observed behavior

When sending a message with an uploaded image:

- the plugin can see that the thread later contains an image `file` part
- but there is no documented way to inject/modify `parts`
- modifying `output.content` does not produce a visible injected description for the sent image message
- the assistant still behaves as if no image was attached

In practice, this means transform plugins cannot reliably operate on image-bearing outgoing messages.

## Expected behavior

At least one of these should exist:

1. `chat.message.willSend` should expose multipart-aware input/output, e.g. `parts`, `attachments`, or equivalent
2. There should be a separate image/message transform hook specifically for multipart messages
3. The docs should explicitly describe whether modifying `output.content` is expected to affect image-bearing messages, and how

## Suggested API direction

Something like:

```ts
interface ChatMessagePart {
type: 'text' | 'file' | ...
text?: string
url?: string
mediaType?: string
filename?: string
}

interface ChatMessageWillSendInput {
threadId: string
content: string
parts?: ChatMessagePart[]
model: string
providerId: string
}

interface ChatMessageWillSendOutput {
content?: string
parts?: ChatMessagePart[]
cancel?: boolean
}
```

Even read-only `parts` would already help a lot for plugin authors.

## Repro use case

A minimal repro plugin idea:

- register `chat.message.willSend`
- detect uploaded image in outgoing message
- append a generated image description before sending

This is currently blocked by the lack of documented multipart/message-part support in the hook API.

Thanks — this would unlock a whole class of useful transform/composite plugins.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.