deepjavalibrary / deepjavalibrary/djl-serving
[LMI] JVM front-end heap fixed at 1 GB; insufficient for advanced async handler patterns
- Dominant language
- Java
- Stars
- 253
- Forks
- 96
- Avg merge
- 23h 6m
- Merged PRs (30d)
- 3
Description
## Description
The DJL Serving JVM front-end launches with a **hard-coded 1 GB heap** (`-Xmx1g -Xms1g`), baked into the Dockerfile `JAVA_OPTS` env var for every container variant:
- `serving/docker/lmi.Dockerfile:31`
- `serving/docker/tensorrt-llm.Dockerfile:27`
- `serving/docker/Dockerfile:42`
- `serving/docker/pytorch-gpu.Dockerfile:46`
- `serving/docker/aarch64.Dockerfile:23`
While `JAVA_OPTS` is technically overridable as a container environment variable (documented in `serving/docs/configurations_global.md:161`), this requires users to know about it and set it at the container/IC level — there is no `serving.properties` option or `option.*` property to control the heap. For SageMaker deployments, users must pass `JAVA_OPTS` in the IC container `Environment`, which is not discoverable from the LMI configuration docs.
## Problem
Under advanced serving patterns — particularly custom `model.py` handlers that perform async operations alongside the vLLM engine in rolling-batch mode — the 1 GB heap can be exhausted by DJL's async response-tracking infrastructure (e.g., `ChunkedBytesSupplier`, `AsyncRequestManager` response queues). When this happens, the failure mode is a hard kill:
```
Terminating due to java.lang.OutOfMemoryError: Java heap space
```
This terminates the entire JVM (due to `-XX:+ExitOnOutOfMemoryError`), kills the Python worker, and forces a full container restart.
## Proposal
Consider one or more of:
1. **Scale the default heap with instance size.** On instances with 32+ GB RAM, 1 GB for the Java front-end is conservative. A heuristic default (e.g., `min(4GB, 10% of RAM)`) would be more appropriate for production workloads.
2. **Expose a **`serving.properties`** option** (e.g., `option.jvm_heap_size` or `serving.jvm_max_heap`) so users can tune the heap without needing to know about `JAVA_OPTS`. This would parallel the existing `option.*` convention for other tuning knobs.
3. **Document the **`JAVA_OPTS`** override path** more prominently in the LMI deployment guide — currently it's only mentioned in the global config reference, not in the LMI-specific docs where users deploying custom handlers would look.
Will this change the current api? No — the default behavior (1 GB heap) would remain unchanged. The proposal adds an optional configuration knob (e.g., a serving.properties option) for users who need to tune it.
How? Expose an optional serving.properties option (e.g., option.jvm_heap_size) and/or scale the default heap with available instance memory, so advanced handler patterns aren't constrained by the hard-coded 1 GB ceiling.
Who will benefit from this enhancement? Customers running custom model.py handlers with advanced async patterns (multi-step inference, adapter chaining, orchestration logic) on instances with ample CPU memory, where the 1 GB default is the binding constraint rather than physical RAM.
Contributor guide
Research direction
Start by comparing the JAVA_OPTS definitions in serving/docker/lmi.Dockerfile, tensorrt-llm.Dockerfile, Dockerfile, pytorch-gpu.Dockerfile, and aarch64.Dockerfile with the configuration described in serving/docs/configurations_global.md. Trace how serving.properties and option.* settings are handled, then review the LMI deployment documentation. Done should mean a decided heap configuration path is implemented or documented without changing the default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java
- Domain
- backend, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100