deepjavalibrary / deepjavalibrary/djl-serving

[LMI] JVM front-end heap fixed at 1 GB; insufficient for advanced async handler patterns

Open
#3,078 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
253
Forks
96
Avg merge
23h 6m
Merged PRs (30d)
3

Description

## Description

The DJL Serving JVM front-end launches with a **hard-coded 1 GB heap** (`-Xmx1g -Xms1g`), baked into the Dockerfile `JAVA_OPTS` env var for every container variant:

- `serving/docker/lmi.Dockerfile:31`
- `serving/docker/tensorrt-llm.Dockerfile:27`
- `serving/docker/Dockerfile:42`
- `serving/docker/pytorch-gpu.Dockerfile:46`
- `serving/docker/aarch64.Dockerfile:23`

While `JAVA_OPTS` is technically overridable as a container environment variable (documented in `serving/docs/configurations_global.md:161`), this requires users to know about it and set it at the container/IC level — there is no `serving.properties` option or `option.*` property to control the heap. For SageMaker deployments, users must pass `JAVA_OPTS` in the IC container `Environment`, which is not discoverable from the LMI configuration docs.

## Problem

Under advanced serving patterns — particularly custom `model.py` handlers that perform async operations alongside the vLLM engine in rolling-batch mode — the 1 GB heap can be exhausted by DJL's async response-tracking infrastructure (e.g., `ChunkedBytesSupplier`, `AsyncRequestManager` response queues). When this happens, the failure mode is a hard kill:

```
Terminating due to java.lang.OutOfMemoryError: Java heap space

```

This terminates the entire JVM (due to `-XX:+ExitOnOutOfMemoryError`), kills the Python worker, and forces a full container restart.

## Proposal

Consider one or more of:

1. **Scale the default heap with instance size.** On instances with 32+ GB RAM, 1 GB for the Java front-end is conservative. A heuristic default (e.g., `min(4GB, 10% of RAM)`) would be more appropriate for production workloads.
2. **Expose a **`serving.properties`** option** (e.g., `option.jvm_heap_size` or `serving.jvm_max_heap`) so users can tune the heap without needing to know about `JAVA_OPTS`. This would parallel the existing `option.*` convention for other tuning knobs.
3. **Document the **`JAVA_OPTS`** override path** more prominently in the LMI deployment guide — currently it's only mentioned in the global config reference, not in the LMI-specific docs where users deploying custom handlers would look.

Will this change the current api? No — the default behavior (1 GB heap) would remain unchanged. The proposal adds an optional configuration knob (e.g., a serving.properties option) for users who need to tune it.

How? Expose an optional serving.properties option (e.g., option.jvm_heap_size) and/or scale the default heap with available instance memory, so advanced handler patterns aren't constrained by the hard-coded 1 GB ceiling.

Who will benefit from this enhancement? Customers running custom model.py handlers with advanced async patterns (multi-step inference, adapter chaining, orchestration logic) on instances with ample CPU memory, where the 1 GB default is the binding constraint rather than physical RAM.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the JAVA_OPTS definitions in serving/docker/lmi.Dockerfile, tensorrt-llm.Dockerfile, Dockerfile, pytorch-gpu.Dockerfile, and aarch64.Dockerfile with the configuration described in serving/docs/configurations_global.md. Trace how serving.properties and option.* settings are handled, then review the LMI deployment documentation. Done should mean a decided heap configuration path is implemented or documented without changing the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
backend, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.