MemberJunction / MemberJunction/MJ

Issue: Image/Media Attachments in Conversation Need Placeholder System

Open
#1,810 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Problem Statement

There are two related issues with how images/media attachments are handled in agent conversations:

### Issue 1: LLMs Truncate Base64 Data in Outputs

When Sage (or any agent) needs to pass an image from the conversation to an action like `Generate Image` with `SourceImage` parameter, the LLM cannot reliably copy the full base64 data. Base64 images are typically 500KB-2MB+ of text, and LLMs naturally truncate such massive strings in their outputs.

**Example of truncated output from Sage:**
```json
{
"actions": [
{
"name": "Generate Image",
"params": {
"Prompt": "Transform the man to a cowboy...",
"SourceImage": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAYAAAByNR6YAAAgAElEQVR4Xu3dB3Mc1R3H8e8f//784iIiIiIiIogqiHv9+/jzP3vnPf5...",
"Model": "Nano Banana Pro"
}
}
]
}
```

The `...` is literally in the LLM output - it truncated the base64 string, making it unusable.

### Issue 2: Token Inefficiency in Conversation Messages

The conversation messages sent to the LLM contain full base64 data, which is extremely token-inefficient:

```json
{
"conversationMessages": [
{
"role": "user",
"content": [
{
"type": "text",
"content": "Sage can you read about this image and generate me something similar?"
},
{
"type": "image_url",
"content": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAm0AAAJyCAIAAAAZ6WCnAAAAIGNIUk0AAHomAACAhAAA+gAAAIDoAAB1MAAA6mAAADqYAAAXcJy6UTwAAAAGYktHRAD/AP8A/6C9p5MAAAABb3JOVAHPoneaAACAAElEQVR42uz995Nl6XnfCT6vPf7atOW72sCQcCRIihqMxNidCEVshEzExowUCsVKG9of9Fco9Iv+jNVsSCNDShqtZikORQokBIIE0A7t0VVdXSarKt3Na48/r3n2h/fmrWwDotFddAPE+4mKjKzMc93Jm/k9j/s+BBHB4/F4PB7Px4J+1k/A4/F4PJ6fY7yOejwej8fz8fE66..."
}
]
}
]
}
```

This wastes tokens and increases costs significantly, especially with multiple images in a conversation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.