deepjavalibrary / deepjavalibrary/djl-serving
[LMI] `enable_lora` unconditionally overrides `maxWorkers` to 1, blocking advanced multi-adapter handler patterns
- Dominant language
- Java
- Stars
- 253
- Forks
- 96
- Avg merge
- 23h 6m
- Merged PRs (30d)
- 3
Description
## Description
When `option.enable_lora=true`, `LmiConfigRecommender.setPropertiesForLora` **unconditionally** overwrites `maxWorkers` to `1`:
```java
// wlm/src/main/java/ai/djl/serving/wlm/LmiConfigRecommender.java, lines 145-159
private static void setPropertiesForLora(Properties lmiProperties) {
// If option.enable_lora=true, set load_on_devices=0 and maxWorkers=1 because we
// only
// support one worker thread
// for LoRA.
// TODO: Support multiple worker threads for LoRA.
boolean enableLora = Boolean.parseBoolean(lmiProperties.getProperty("option.enable_lora"));
if (enableLora) {
logger.info(
"option.enable_lora is set to true, setting load_on_devices=0 and"
+ " maxWorkers=1");
lmiProperties.setProperty("load_on_devices", "0");
lmiProperties.setProperty("maxWorkers", "1");
}
}
```
Two properties of this code are noteworthy:
1. **The override is unconditional.** It uses `setProperty`, not `setDefault` or a check-then-set pattern. Even if a user explicitly sets `maxWorkers` in `serving.properties`, it is silently overwritten to `1`. There is no way for a user to opt out.
2. **The restriction is acknowledged as temporary.** The inline `TODO: Support multiple worker threads for LoRA` indicates this is a known limitation intended to be lifted, not a fundamental constraint of LoRA serving. The vLLM `AsyncLLMEngine` itself is designed for concurrent requests (e.g., `max_rolling_batch_size=256`).
## Problem
`maxWorkers=1` means the container can process exactly **one DJL-tracked request at a time** through the Python worker. This blocks use cases where a custom `model.py` handler needs to perform **multiple sequential adapter invocations within a single request** — a pattern needed for multi-adapter chaining (e.g., two LoRA adapters run back-to-back on a single input, each with its own pre/post-processing).
The natural implementation of such a handler would issue sub-requests to the container's own `/invocations` endpoint (one per adapter stage), relying on the standard DJL request path to resolve the LoRA via the `"adapters"` payload key (`async_utils._extract_lora_adapter`, line ~120) and get clean request tracking. However, with `maxWorkers=1`, the single worker is occupied by the parent request and cannot service the sub-request — resulting in a **self-invocation deadlock**:
1. Parent request enters `handle()`, issues localhost POST to `127.0.0.1:8080/invocations`
2. Sub-request arrives at the same container, but no worker is free
3. DJL's `InferenceRequestHandler` times out (`Read chunk timeout` / `ChunkedBytesSupplier`)
4. Connection is reset → `ConnectionResetError: [Errno 104] Connection reset by peer` in the parent
The parent request then fails with a 424, and if the timeout cascades, the worker is killed and reloaded.
## Proposal
**Support multiple worker threads for LoRA.** As the inline `TODO` in `setPropertiesForLora` already notes, this is intended to be lifted. vLLM's engine handles concurrent LoRA requests through its own scheduling (adapter-aware KV cache, LRU adapter swapping, `max_loras` GPU slots). The `maxWorkers` restriction predates the current maturity of vLLM's LoRA support and may no longer be necessary.
Lifting this restriction would unblock multi-adapter chaining patterns where a custom handler issues sequential sub-requests (one per adapter stage) to the container's own endpoint — each cleanly tracked by DJL's request accounting — without hitting the self-invocation deadlock described above.
Will this change the current api? Yes
How? Allows more than one concurrent DJL worker thread when LoRA is enabled, changing the default concurrency model for LoRA endpoints.
Who will benefit from this enhancement? Customers building custom handlers that perform multiple adapter invocations per request (e.g., multi-stage LoRA chaining) within a single container.
Contributor guide
Research direction
Start in wlm/src/main/java/ai/djl/serving/wlm/LmiConfigRecommender.java, especially setPropertiesForLora around lines 145-159, and trace how maxWorkers is applied when option.enable_lora=true. Read the described async_utils._extract_lora_adapter path and the /invocations request flow to understand nested adapter requests. Done means LoRA endpoints can support multiple workers without silently overwriting an explicitly configured maxWorkers, while preserving request handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100