deepjavalibrary / deepjavalibrary/djl-serving
vllm_async_service: Inject custom output formatters into VLLMHandler
- Dominant language
- Java
- Stars
- 253
- Forks
- 96
- Avg merge
- 23h 6m
- Merged PRs (30d)
- 3
Description
## Description
Hello,
Currently, the method `preprocess_request()` in `VLLMHandler` (vllm_async_service.py) initializes predefined stream and non stream output formatters. When using vllm_async_service as entry point in AWS LMI containers, defining a custom_output_formatter in model.py (appropriately decorated with @output_formatter) would not overwrite existing output formatters set by the service.
Current limitations: The functionality of specifying a custom output formatter is limited to text generation only as stated in the documentation:
> TextGenerationOutput: This subclass of RequestOutput is specific to text generation tasks. Right now this is the only task supported for custom output formatter. Each text generation task can generate multiple sequences.
The output formatters utilized by the current async service operate on a richer set of protocols such as ChatCompletionResponse and CompletionResponse.
Will this change the current api? How?
Probably the api will need to be adapted to accept user supplied formatters.
Who will benefit from this enhancement?
Users who want to have a finer control over the service output
## References
- list reference and related literature
- list known implementations
Contributor guide
Research direction
Start with vllm_async_service.py, especially VLLMHandler.preprocess_request(), and review how model.py defines custom_output_formatter with @output_formatter. Compare the existing stream and non-stream formatters with the ChatCompletionResponse and CompletionResponse protocols. Done means a user-supplied formatter can replace the service defaults without breaking supported response paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100