[Usage]: pytorch workflow beam search

System Info

Python 3.12.3
Name: tensorrt_llm
Version: 1.1.0rc1

How would you like to use TensorRT-LLM

Question 1

How to use logits processors in beam search? I found that the logits processor only works on one sequence, not on best_of sequences.

Question 2

If max_beam_width is specified when the engine starts, does it mean that beam search must be used, and that even ordinary requests(use_beam_search=False in SamplingParams) will throw an error:

File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/executor_request_queue.py", line 464, in _validate_and_filter_requests
AssertionError: Request beam width 1 is not equal to max_beam_width 8. This is not supported!

Question 3

In my case, max_tokens set to 20-30, it works ok, but when max_tokens reach 60, it will crash.

 File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 283, in _event_loop_wrapper
    raise e
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 279, in _event_loop_wrapper
    self.event_loop()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 967, in _executor_loop
    profile_step()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 516, in profile_step
    start_event_2.record()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/streams.py", line 187, in record
    super().record(stream)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/pytorch/pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f63cd9c4568 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x55 (0x7f63cd960221 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3e2 (0x7f63cda81c82 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0xbae8fb (0x7f63ce67c8fb in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xba98cb (0x7f63ce6778cb in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xbb1084 (0x7f63ce67f084 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0x453c52 (0x7f643143ac52 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f63cd9a1409 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x70d688 (0x7f64316f4688 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x70dab0 (0x7f64316f4ab0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)
...

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start with _torch/pyexecutor/executor_request_queue.py around request validation and _torch/pyexecutor/py_executor.py around profile_step and the executor loop. Reproduce the reported beam-search settings with max_beam_width 8, use_beam_search=False, and max_tokens near 60, then isolate whether the request assertion and CUDA illegal-memory-access failure are related. Done means the supported behavior and a reproducible resolution or clear limitation are documented.