deepjavalibrary / deepjavalibrary/djl

TensortRT-LLM compilation parameter overwrite

Open
#3,489 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

## Description

When deploying a Mistral Instruct 7B v.02 on a SageMaker endpoint (ml.g5.12xlarge) using the TensortRT-LLM backend (just-in-time compilation), I noticed that some of the serving parameters get overwritten.

Specifically, I used the following set of serving properties:
"SERVING_ENGINE": "MPI",
"OPTION_TENSOR_PARALLEL_DEGREE": "1",
"OPTION_MAX_ROLLING_BATCH_SIZE": "16",
"OPTION_ROLLING_BATCH":"trtllm",
"OPTION_MAX_INPUT_LEN":"2048",
"OPTION_MAX_OUTPUT_LEN":"16",
"OPTION_BATCH_SCHEDULER_POLICY": "max_utilization"

CouldWatch logs state the following:

- max_input_len is 2048 is larger than max_seq_len 16, clipping it to max_seq_len
- max_num_tokens (256) shouldn't be greater than max_seq_len * max_batch_size (256), specifying to max_seq_len * max_batch_size (256).

max_num_tokens is marked in the [documentation](https://docs.djl.ai/master/docs/serving/serving/docs/lmi/user_guides/trt_llm_user_guide.html#advanced-tensorrt-llm-configurations) as taking the default value 16384

### Expected Behavior
Parameters to preserve their supplied value

### Error Message
When submitting inference requests:
`this model is compiled to take up to 16 tokens. But actual tokens is 987 > 16. Please set with option.max_input_len=987`

## How to Reproduce?
Recipe to reproduce the error presented in description

## Environment Info

Docker image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/djl-inference:0.29.0-tensorrtllm0.11.0-cu124

```
PASTE OUTPUT HERE
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.