NVIDIA / NVIDIA/TensorRT-LLM

Lookahead decoding and multiple profiles support

Open
#3,666 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Hi,

After #3137 was solved, lookahead decoding with multimodal inputs seems to work correctly, but only when multiple_profiles==disabled. When it's enabled I get the following error:

[TensorRT-LLM][ERROR] ICudaEngine::getIOTensorName: Error Code 3: API Usage Error (Parameter check failed, condition: index >= 0 && index < getNbIOTensors(). Invalid IO Tensor Index: 29. Valid range of IO Tensor Indexis [0, 26). Upper bound of IO Tensor Index can be queried with ICudaEngine::getNbIOTensors.)
[TensorRT-LLM][ERROR] IExecutionContext::inferShapes: Error Code 3: API Usage Error (Parameter check failed, condition: validDimension. Supplied binding dimensions [1] for tensor (getIOTensorName returned nullptr) exceed min..max range for profile 1. Valid range is [64]..[128].)
[TensorRT-LLM][ERROR] Encountered an error in forwardAsync function: Invalid input shape (/home/jenkins/agent/workspace/LLM/release-0.18/L0_Test-x86_64/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:574)
1       0x7fd1e98c82b7 /home/maximilianolevi/.cache/pypoetry/virtualenvs/tensorrt-inference-8MUMp6os-py3.10/lib/python3.10/site-packages/tensorrt_llm/libs/libtensorrt_llm.so(+0x6e52b7) [0x7fd1e98c82b7]
2       0x7fd1ea5ac458 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::prepareBuffers(std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, int) + 184
3       0x7fd1ea5b59a6 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::executeStep(std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, int) + 1510
4       0x7fd1ea5b628f tensorrt_llm::batch_manager::TrtGptModelInflightBatching::executeBatch(tensorrt_llm::batch_manager::ScheduledRequests const&) + 223
5       0x7fd1ea5bfb7a tensorrt_llm::batch_manager::TrtGptModelInflightBatching::forwardAsync(std::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&) + 1802
6       0x7fd1ea64c775 tensorrt_llm::executor::Executor::Impl::forwardAsync(std::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > >&) + 437
7       0x7fd1ea6584e6 tensorrt_llm::executor::Executor::Impl::executionLoop() + 1206
8       0x7fd4f2a295c0 /home/maximilianolevi/.cache/pypoetry/virtualenvs/tensorrt-inference-8MUMp6os-py3.10/lib/python3.10/site-packages/torch/lib/libtorch.so(+0x145c0) [0x7fd4f2a295c0]
9       0x7fd4fddedea7 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7fd4fddedea7]
10      0x7fd4fdf03acf clone + 63

To reproduce:

  • Enable lookahead decoding
  • Compile with multiple_profiles enabled
  • Request has multimodal input (e.g. just custom prompt table with fake vocabulary)
  • batch size > 1
  • Inflight fused batching is enabled
  • Model is Llama 8B.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with lookahead decoding, multiple_profiles, multimodal input, batch size greater than 1, inflight fused batching, and the Llama 8B model. Start in tllmRuntime.cpp around line 574 and inspect TrtGptModelInflightBatching::prepareBuffers; done means this configuration completes without invalid tensor-index or input-shape errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, pytorch
Domain
backend, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.