deepjavalibrary / deepjavalibrary/djl-serving

vllm_async_service: Inject custom output formatters into VLLMHandler

Open
#2,819 4 comments 1 reaction 0 assignees View on GitHub
enhancement stale
Dominant language
Java
Stars
253
Forks
96
Avg merge
23h 6m
Merged PRs (30d)
3

Description

## Description
Hello,

Currently, the method `preprocess_request()` in `VLLMHandler` (vllm_async_service.py) initializes predefined stream and non stream output formatters. When using vllm_async_service as entry point in AWS LMI containers, defining a custom_output_formatter in model.py (appropriately decorated with @output_formatter) would not overwrite existing output formatters set by the service.

Current limitations: The functionality of specifying a custom output formatter is limited to text generation only as stated in the documentation:

> TextGenerationOutput: This subclass of RequestOutput is specific to text generation tasks. Right now this is the only task supported for custom output formatter. Each text generation task can generate multiple sequences.

The output formatters utilized by the current async service operate on a richer set of protocols such as ChatCompletionResponse and CompletionResponse.

Will this change the current api? How?

Probably the api will need to be adapted to accept user supplied formatters.

Who will benefit from this enhancement?

Users who want to have a finer control over the service output

## References
- list reference and related literature
- list known implementations

Contributor guide

Open the contributing guide

Research direction

Start with vllm_async_service.py, especially VLLMHandler.preprocess_request(), and review how model.py defines custom_output_formatter with @output_formatter. Compare the existing stream and non-stream formatters with the ChatCompletionResponse and CompletionResponse protocols. Done means a user-supplied formatter can replace the service defaults without breaking supported response paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.