deepjavalibrary / deepjavalibrary/djl
Unable to load model with LoRA adapter using DJL image with SageMaker fast model loader
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 760
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 17
Description
## Description
When attempting to load **Meta-Llama-3.1-8B-Instruct-AWQ-INT4** with a **LoRA adapter** using the **DJL image** with **SageMaker fast model loader**, the model fails to load properly.
### Expected Behavior
**The model should load successfully** with the LoRA adapter applied, allowing for inference with the adapted model weights.
### Error Message
```text
[INFO ] PyProcess - W-140-model-stdout: [1,0]:RuntimeError: Number of chunk files and slices do not match.
```
```text
2025-03-12T05:35:42.978Z
Caused by: ai.djl.engine.EngineException: Failed to initialize model: invoke handler failure
2025-03-12T05:36:05.015Z [INFO ] PyProcess - W-140-model-stdout: [1,0]:INFO 03-12 05:36:05 loader.py:27] SageMakerFastModelLoader loading shard for tp_rank: 0, pp_rank: 0
[INFO ] PyProcess - W-140-model-stdout: [1,0]:INFO 03-12 05:36:05 loader.py:27] SageMakerFastModelLoader loading shard for tp_rank: 0, pp_rank: 0
2025-03-12T05:36:05.194Z [INFO ] PyProcess - W-140-model-stdout: [1,0]:**ERROR::Failed invoke service.invoke_handler()*| [INFO ] PyProcess - W-140-model-stdout: [1,0]:**ERROR::Failed invoke service.invoke_handler()
```
---
## How to Reproduce?
1. Shard the model using SageMaker Studio by running an optimization job, and place the adapter in the `adapters` folder where the model is located.
2. Create an endpoint with the **DJL image** and the model artifacts.
3. Add these configurations in `serving.properties`:
4. Check the error logs in cloudwatch.
```properties
option.load_format=sagemaker_fast_model_loader
option.tensor_parallel_degree=1
option.max_model_len=16000
serving.max_model_len=16000
option.enable_lora=true
option.max_loras=10
```
---
## What Have You Tried to Solve It?
- Followed the folder structure and modified `serving.properties` as mentioned in the official DJL documentation:
**[[Multi-LoRA Adapter Inference Guide](https://docs.djl.ai/master/docs/demos/aws/sagemaker/large-model-inference/sample-llm/multi_lora_adapter_inference.html)](https://docs.djl.ai/master/docs/demos/aws/sagemaker/large-model-inference/sample-llm/multi_lora_adapter_inference.html)**
- Despite these steps, the model still fails to load successfully.
---
## Additional Information
- **DJL Image Version**: 763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.31.0-lmi13.0.0-cu124
Contributor guide
Research direction
Start with the SageMakerFastModelLoader shard-loading path mentioned in the logs and the serving.properties settings, then compare them with the Multi-LoRA Adapter Inference Guide. Use the CloudWatch reproduction with the DJL image version 0.31.0-lmi13.0.0-cu124; done means the model loads with the LoRA adapter applied without the chunk-file and slice mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, machine-learning
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100