Bulk processing of inputs in an LLM - bulk categorization

Currently running prompt flow the "input" is defined as a single "item".

Currently for the classification example it can only classify 1 input at a time.

There needs to be a feature to be able to put in many products at once so that a single prompt can output many categorizations.

To be able to import many inputs at once, and have them all output and linked to original input to make sure that accuracy can be traced.

This way the "cost" of the prompt tokens to explain what must be done can be "amortized" across many inputs.

The same way in an eval you can "bulk" process inputs. The same must be done with a general flow.

Start by comparing the requested general-flow behavior with the existing bulk-processing behavior in evals, then trace how flow inputs and outputs are represented. Done means a flow can import many products, produce a categorization for each, link every result to its original input for accuracy tracing, and amortize prompt-instruction tokens across the batch.