Performance issue in vllm async generation

I noticed a potential performance issue in the current async generation implementation in vllm_generation.py, especially when multiple workers are available.
## Current behavior
For synchronous generation, the implementation shards the input batch evenly across workers, runs generation on all workers in parallel, and then synchronizes and merges the results. This generally leads to good hardware utilization.
However, for asynchronous generation, the behavior seems suboptimal in multi-worker settings. In the current implementation, for a given batch, async generation only dispatches the generation task to a single worker, while the remaining workers stay idle. As a result, when dp > 1, most workers do not participate in generation at all, leading to significant resource underutilization.
This is especially problematic for workloads with long contexts or highly variable generation lengths, which are common in RL rollouts.

## Expected / ideal behavior
If we do not require each sample to be generated by a specific worker, a more efficient async strategy would resemble a producer–consumer (work-stealing) model:
* The input batch is first split into multiple microbatches
* All workers run asynchronously
* Each worker repeatedly pulls a microbatch from a shared queue, performs generation, and then immediately pulls the next available microbatch
* This continues until the entire batch is processed, after which results are merged back using the original indices
This allows faster workers to process more microbatches and mitigates straggler effects caused by long or uneven sequences.

## Empirical results
I tested this idea using Skywork-OR1-7B on DAPO-Math, with the following setup:
* Hardware: 8 × A800 40GB
* generation.vllm_cfg.max_model_len = 18432
* eval.num_tests_per_prompt = 16
* Parallelism: pp = 1, ep = 1, tp = 1, dp = 8
### Results:
* Synchronous generation (generation.vllm_cfg.async_engine = false):
* 65,024 rollouts generated in ~48.5 hours
* Baseline throughput
* Current async generation (generation.vllm_cfg.async_engine = true):
* Only a single worker is active per batch
* Other workers remain idle, resulting in much lower utilization
* Async generation with work-stealing microbatch scheduling (producer–consumer style):
* 74,752 rollouts generated in ~33.5 hours
* **~67% speedup** compared to synchronous generation
* Estimated ~7× faster than the current async implementation due to full worker participation

## Note
* This approach changes execution order across workers, so strict bitwise determinism may not be preserved unless per-sample seeding is used