Different generation parameters in the same batch

Hello team,

Today, batch generation works like the HF generate() function: it accepts several input texts but generation parameters (like temperature, top k, etc.) apply to the whole batch, so it is not possible to use different parameters within the same batch.

Is it because using different parameters in the same batch would degrade performance so much that it would defeat the purpose of batch generation?

Ideally it would be awesome if one could do something like this:

generator.generate_batch([
    {"input":input_1, "max_length":30, "sampling_topk":10},
    {"input":input_2, "max_length":150, "sampling_topk":50},
    {"input":input_3, "max_length":10, "sampling_topk":50},
    ...
])

For example this is something that can be achieved with NVIDIA Faster Transformers: https://github.com/NVIDIA/FasterTransformer/blob/main/examples/pytorch/gpt/gpt_example.py

Thank you!

Start by tracing the batch-generation entry point behind generator.generate_batch and compare its current parameter handling with the per-input example in the issue. Use the FasterTransformer reference to investigate whether per-item max_length and sampling parameters can be supported, and define completion around accepting the proposed input shape without unacceptable batch-performance degradation.