[GRPO] NCCL timeout when GRPO training with vllm, but get "ValueError: The decoder prompt (length 5951) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens. "

### Reproduction

```
Traceback (most recent call last):
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/trl/scripts/vllm_serve.py", line 303, in llm_worker
result = method(*args, **kwargs)
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/utils.py", line 1196, in inner
return fn(*args, **kwargs)
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 465, in generate
self._validate_and_add_requests(
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1354, in _validate_and_add_requests
self._add_request(
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1372, in _add_request
self.llm_engine.add_request(
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 183, in add_request
prompt_str, request = self.processor.process_inputs(
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 240, in process_inputs
self._validate_model_inputs(processed_inputs, lora_request)
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 331, in _validate_model_inputs
self._validate_model_input(decoder_inputs,
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 380, in _validate_model_input
raise ValueError(
ValueError: The decoder prompt (length 5951) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than t
he number of text tokens.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/process.py", line 317, in _bootstrap
util._exit_function()
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
p.join()
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/popen_fork.py", line 43, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
^CException ignored in:
Traceback (most recent call last):
File "/opt/conda/envs/easyr1/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 319, in __call__
core_engine.close()
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 295, in close
proc_handle.shutdown()
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/utils.py", line 128, in shutdown
self._finalizer()
File "/opt/conda/envs/easyr1/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/opt/conda/envs/easyr1/lib/python3.10/site-packages/vllm/v1/utils.py", line 137, in shutdown
proc.join(5)
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/popen_fork.py", line 40, in wait
if not wait([self.sentinel], timeout):
File "/opt/conda/envs/easyr1/lib/python3.10/multiprocessing/connection.py", line 931, in wait
ready = selector.select(timeout)
File "/opt/conda/envs/easyr1/lib/python3.10/selectors.py", line 416, in select
fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt:
```

I used grpo and vllm for training. After about 3000 steps of training, vllm got stuck and the main program timed out. When I terminated the vllm program by pressing ctrl+c, I got this error. Is there a bug in vllm's length truncation? If an error occurs, why does it not exit directly but enter a deadlock and time out?

I didn't get similar insights in previous issue https://github.com/huggingface/trl/issues/2923.

### System Info

trl 0.16.0
vllm 0.8.5.post1
torch 2.6.0+cu124
ubuntu 18.04
NVIDIA A6000

### Checklist

- [x] I have checked that my issue isn't already filed (see [open issues](https://github.com/huggingface/trl/issues?q=is%3Aissue))
- [x] I have included my system information
- [x] Any code provided is minimal, complete, and reproducible ([more on MREs](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any code provided is properly formatted in code blocks, (no screenshot, [more on code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks))
- [x] Any traceback provided is complete