deepjavalibrary / deepjavalibrary/djl-demo

trtllm_rollingbatch_deploy_llama_13b.ipynb notebook issue

Open
#424 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
352
Forks
139
PR merge metrics
No merged PRs in 30d

Description

Following example: https://github.com/deepjavalibrary/djl-demo/blob/2a5152f578f5954b8b68acdee18eed4e2a75c81f/aws/sagemaker/large-model-inference/sample-llm/trtllm_rollingbatch_deploy_llama_13b.ipynb, errors out with a health check error with current code, the following cell needs to be uncommented for successful deployment:

```
model.deploy(initial_instance_count=1,
instance_type=instance_type,
endpoint_name=endpoint_name,
container_startup_health_check_timeout=3600 ##uncomment
)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.