deepjavalibrary / deepjavalibrary/djl-demo
trtllm_rollingbatch_deploy_llama_13b.ipynb notebook issue
Open
- Dominant language
- Jupyter Notebook
- Stars
- 352
- Forks
- 139
- PR merge metrics
- No merged PRs in 30d
Description
Following example: https://github.com/deepjavalibrary/djl-demo/blob/2a5152f578f5954b8b68acdee18eed4e2a75c81f/aws/sagemaker/large-model-inference/sample-llm/trtllm_rollingbatch_deploy_llama_13b.ipynb, errors out with a health check error with current code, the following cell needs to be uncommented for successful deployment:
```
model.deploy(initial_instance_count=1,
instance_type=instance_type,
endpoint_name=endpoint_name,
container_startup_health_check_timeout=3600 ##uncomment
)
```
Contributor guide
Assessment
This issue has not been assessed yet.