microsoft / microsoft/promptflow
[Feature Request] Multi-Modality Support for More Comprehensive Applications
@lisagreenview is already working on this.
Since Sep 14, 2023.
- Dominant language
- Python
- Stars
- 11.2k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
While Prompt Flow already offers impressive flexibility regarding data types within the overall stack, encompassing flow input/output and node intermediate data, it is crucial to address the limited support for visualization and tracing, which primarily revolves around text-oriented capabilities.
To further empower developers in creating comprehensive multi-modality applications, the following enhancements are highly recommended, although not exclusively limited to:
-
Media Type Specification: Extend the functionality of Prompt Flow to enable users to specify the media type (e.g., image, video, audio) for flow input/output, facilitating appropriate visualization and interpretation.
-
Expanded Testing/Chat Interface: Enhance the test/chat interface to accommodate inputting varying media types, ensuring seamless interaction with diverse modalities throughout the development process.
-
Media Type Specification for Node Intermediate Data: Provide developers with the ability to specify the media type for node intermediate data and ensure proper visualization within the Visual Studio Code (VSC) extension and portal, promoting a more integrated and holistic development experience.
-
Streaming Output for Multimedia Types: Particularly for media types such as video and audio, enable the streaming output capability, allowing for real-time and dynamic visualization that caters to the specific demands of multimedia applications.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.