Plugin API limitation: chat.message.willSend only exposes content, not multipart image parts
- Dominant language
- TypeScript
- Stars
- 53
- Forks
- 17
- Avg merge
- 2h 5m
- Merged PRs (30d)
- 1
Description
## Summary
`chat.message.willSend` is currently documented as a text-only hook (`input.content` / `output.content`), but real user messages with uploaded images are stored as multipart `message.parts[]` with `type: "file"` image entries.
This makes it effectively impossible to build a transparent image-aware transform plugin (for example, a vision proxy that auto-describes images for non-vision models) using the current public plugin API.
## Why this matters
I was trying to build an Alma plugin similar to `pi-vision-proxy`:
- user sends text + image
- plugin intercepts before send
- plugin routes the image to a vision-capable model
- plugin injects the generated description back into the outgoing message
- non-vision models can then “see” the image through the injected description
This workflow depends on being able to inspect and modify image-bearing outgoing messages.
## What the docs currently say
The docs for `chat.message.willSend` define the hook as:
```ts
interface ChatMessageWillSendInput {
threadId: string;
content: string;
model: string;
providerId: string;
}
interface ChatMessageWillSendOutput {
content?: string;
cancel?: boolean;
}
```
So the documented mutation surface is text-only.
## What real thread data looks like
However, when a user uploads an image, the actual stored message shape is multipart and uses `parts`:
```json
{
"message": {
"role": "user",
"parts": [
{
"type": "text",
"text": "这是什么照片"
},
{
"type": "file",
"mediaType": "image/jpeg",
"url": "http://localhost:23001/api/gallery/cache/upload-...jpg",
"filename": "sport_02_with_id.jpg"
}
]
}
}
```
So there is a mismatch:
- real message structure: `message.parts[]`
- public hook mutation surface: `content` only
## Observed behavior
When sending a message with an uploaded image:
- the plugin can see that the thread later contains an image `file` part
- but there is no documented way to inject/modify `parts`
- modifying `output.content` does not produce a visible injected description for the sent image message
- the assistant still behaves as if no image was attached
In practice, this means transform plugins cannot reliably operate on image-bearing outgoing messages.
## Expected behavior
At least one of these should exist:
1. `chat.message.willSend` should expose multipart-aware input/output, e.g. `parts`, `attachments`, or equivalent
2. There should be a separate image/message transform hook specifically for multipart messages
3. The docs should explicitly describe whether modifying `output.content` is expected to affect image-bearing messages, and how
## Suggested API direction
Something like:
```ts
interface ChatMessagePart {
type: 'text' | 'file' | ...
text?: string
url?: string
mediaType?: string
filename?: string
}
interface ChatMessageWillSendInput {
threadId: string
content: string
parts?: ChatMessagePart[]
model: string
providerId: string
}
interface ChatMessageWillSendOutput {
content?: string
parts?: ChatMessagePart[]
cancel?: boolean
}
```
Even read-only `parts` would already help a lot for plugin authors.
## Repro use case
A minimal repro plugin idea:
- register `chat.message.willSend`
- detect uploaded image in outgoing message
- append a generated image description before sending
This is currently blocked by the lack of documented multipart/message-part support in the hook API.
Thanks — this would unlock a whole class of useful transform/composite plugins.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.