microsoft / microsoft/promptflow

[Feature Request] Multi-Modality Support for More Comprehensive Applications

Open
#368 2 comments 1 reaction 1 assignee View on GitHub

@lisagreenview is already working on this.

Since Sep 14, 2023.

enhancement long-term roadmap
Dominant language
Python
Stars
11.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

While Prompt Flow already offers impressive flexibility regarding data types within the overall stack, encompassing flow input/output and node intermediate data, it is crucial to address the limited support for visualization and tracing, which primarily revolves around text-oriented capabilities.

To further empower developers in creating comprehensive multi-modality applications, the following enhancements are highly recommended, although not exclusively limited to:

  • Media Type Specification: Extend the functionality of Prompt Flow to enable users to specify the media type (e.g., image, video, audio) for flow input/output, facilitating appropriate visualization and interpretation.

  • Expanded Testing/Chat Interface: Enhance the test/chat interface to accommodate inputting varying media types, ensuring seamless interaction with diverse modalities throughout the development process.

  • Media Type Specification for Node Intermediate Data: Provide developers with the ability to specify the media type for node intermediate data and ensure proper visualization within the Visual Studio Code (VSC) extension and portal, promoting a more integrated and holistic development experience.

  • Streaming Output for Multimedia Types: Particularly for media types such as video and audio, enable the streaming output capability, allowing for real-time and dynamic visualization that caters to the specific demands of multimedia applications.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.