Lookahead decoding and multiple profiles support
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Hi,
After #3137 was solved, lookahead decoding with multimodal inputs seems to work correctly, but only when multiple_profiles==disabled. When it's enabled I get the following error:
[TensorRT-LLM][ERROR] ICudaEngine::getIOTensorName: Error Code 3: API Usage Error (Parameter check failed, condition: index >= 0 && index < getNbIOTensors(). Invalid IO Tensor Index: 29. Valid range of IO Tensor Indexis [0, 26). Upper bound of IO Tensor Index can be queried with ICudaEngine::getNbIOTensors.)
[TensorRT-LLM][ERROR] IExecutionContext::inferShapes: Error Code 3: API Usage Error (Parameter check failed, condition: validDimension. Supplied binding dimensions [1] for tensor (getIOTensorName returned nullptr) exceed min..max range for profile 1. Valid range is [64]..[128].)
[TensorRT-LLM][ERROR] Encountered an error in forwardAsync function: Invalid input shape (/home/jenkins/agent/workspace/LLM/release-0.18/L0_Test-x86_64/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:574)
1 0x7fd1e98c82b7 /home/maximilianolevi/.cache/pypoetry/virtualenvs/tensorrt-inference-8MUMp6os-py3.10/lib/python3.10/site-packages/tensorrt_llm/libs/libtensorrt_llm.so(+0x6e52b7) [0x7fd1e98c82b7]
2 0x7fd1ea5ac458 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::prepareBuffers(std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, int) + 184
3 0x7fd1ea5b59a6 tensorrt_llm::batch_manager::TrtGptModelInflightBatching::executeStep(std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, std::vector<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&, int) + 1510
4 0x7fd1ea5b628f tensorrt_llm::batch_manager::TrtGptModelInflightBatching::executeBatch(tensorrt_llm::batch_manager::ScheduledRequests const&) + 223
5 0x7fd1ea5bfb7a tensorrt_llm::batch_manager::TrtGptModelInflightBatching::forwardAsync(std::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&) + 1802
6 0x7fd1ea64c775 tensorrt_llm::executor::Executor::Impl::forwardAsync(std::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > >&) + 437
7 0x7fd1ea6584e6 tensorrt_llm::executor::Executor::Impl::executionLoop() + 1206
8 0x7fd4f2a295c0 /home/maximilianolevi/.cache/pypoetry/virtualenvs/tensorrt-inference-8MUMp6os-py3.10/lib/python3.10/site-packages/torch/lib/libtorch.so(+0x145c0) [0x7fd4f2a295c0]
9 0x7fd4fddedea7 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7fd4fddedea7]
10 0x7fd4fdf03acf clone + 63
To reproduce:
- Enable lookahead decoding
- Compile with multiple_profiles enabled
- Request has multimodal input (e.g. just custom prompt table with fake vocabulary)
- batch size > 1
- Inflight fused batching is enabled
- Model is Llama 8B.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with lookahead decoding, multiple_profiles, multimodal input, batch size greater than 1, inflight fused batching, and the Llama 8B model. Start in tllmRuntime.cpp around line 574 and inspect TrtGptModelInflightBatching::prepareBuffers; done means this configuration completes without invalid tensor-index or input-shape errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, pytorch
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100