microsoft / microsoft/simplechat

Support reference images for image generation

Open
#998 5 comments 0 reactions 1 assignee Claimed by @paullizer View on GitHub
enhancement needs_triage
Dominant language
Python
Stars
152
Forks
116
Avg merge
7h 7m
Merged PRs (30d)
122

Description

## Summary
SimpleChat image generation currently generates new images from text-only context. It does not have a way to pass images that were uploaded in chat, referenced from existing conversation context, or found through web search into image generation requests.

## User Value
Users expect modern image generation workflows to accept visual references. This would let them upload an image, reuse an image already in the conversation, or ask the app to locate a relevant public image such as a profile photo, product image, logo, or object reference and use it to guide a newly generated image.

## Proposed Behavior
Image generation should be able to include one or more reference images gathered from supported sources:
- Images uploaded directly in the chat.
- Images already attached to or referenced by the current chat context.
- Images discovered through web search or other retrieval flows when the user asks to use a public image, such as a LinkedIn profile image, company logo, or object reference.
- Reference-image provenance should be visible enough for users/admins to understand what image source was used.
- Unsupported, inaccessible, unsafe, or ambiguous image sources should fail gracefully with clear user guidance.

## Acceptance Criteria
- [ ] Users can provide an uploaded image as context for a new image generation request.
- [ ] Image generation can reuse eligible images from the active chat context.
- [ ] Image generation can use retrieved public image references when the user explicitly asks for them and the source is accessible.
- [ ] The backend validates image source, type, size, accessibility, and policy constraints before sending images to the image generation provider.
- [ ] The UI makes selected reference images clear before generation and allows users to remove them.
- [ ] Tests cover uploaded-image references, chat-context references, retrieved-image references, unsupported images, and provider errors.
- [ ] Documentation explains supported reference-image sources, limitations, and privacy/security expectations.

## Notes
This likely spans chat attachments/context, web retrieval/search flows, image-generation request construction, provider capability detection, frontend selection/preview UX, and security/privacy review for external image retrieval.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.