microsoft / microsoft/promptflow

[Feature Request] Allow any combination of variants to be selected for a run

Open
#944 9 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement long-term
Dominant language
Python
Stars
11.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.

When a flow has more than one node with each node having more than one variant, only one variant can be specified per run. When running with the selected node variant, all other nodes with variants will run with their specified default. Because of this functionality, it is not possible to run all combinations of variants without having to update defaults.

For illustration consider the following flow: I have two LLM nodes each with 2 variants. In each node there is a variant that adds a statement to the prompt to limit the number of characters and the other variant does not. In this situation, I would like to experiment running all combinations of the prompts.
For the illustration, I will refer to node one and two as N1 and N2, and Variant one and to as V1 and V2

Total possible combinations
N1V1*, N2V1*
N1V1*, N2V2
N1V2, N2V1*
N1V2, N2V2
  • Indicates default variant

Given the constraint a node variant can only run against another node's default variant, in the case of multiple nodes with multiple variants, there will exist nodes that are unreachable. In the example above, if we specify V1 as the default for each node. In this case, the combination of non-default variant of Node 1 with non-default variant of Node 2 would not be possible to execute.
 
Describe the solution you'd like
I believe this is probably not something straightforward, but I would think maybe:

  • Provide ability to have variant be defined at the flow level
  • Provide ability to specify which variant will be used for which node (seems that this might get complicated)

Describe alternatives you've considered
I have not been able to find a workaround here aside from changing the default prior to running the experiment.

Additional context
An example of the flow above can be found here:
https://github.com/pragmatical/llmops-promptflow-template/blob/main/web_classification/flows/experiment/flow.dag.yaml

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the example flow at web_classification/flows/experiment/flow.dag.yaml and trace how its node variants are selected for a run. No implementation files or tests are named; the work is done when every combination of variants can be executed without changing defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.