NVIDIA / NVIDIA/TensorRT-Edge-LLM
feat: Add base64 image support to the OpenAI-compatible server
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Detailed description of the requested feature
The experimental OpenAI-compatible server currently supports VLM image inputs only when the image is provided via a local file path. This prevents clients that use the standard OpenAI vision API format from sending images as base64-encoded data URLs.
I would like to add support for base64-encoded images, following the OpenAI-compatible image_url format, e.g. data:image/jpeg;base64,... or data:image/png;base64,....
The server should decode the base64 image data and pass it through the existing VLM image preprocessing/inference pipeline, while retaining the current file-path support for backwards compatibility.
This would make the experimental server more compatible with existing OpenAI-compatible clients and applications, which commonly transmit images as base64 data URLs rather than exposing local filesystem paths. It would also be useful for deployments where the client and inference server do not share the same filesystem.
My implementation is available at: https://github.com/FABallemand/TensorRT-Edge-LLM/tree/feat/server_base64_support
Per CONTRIBUTING.md, I am opening this issue for approval before requesting code review.
Timeline
No hard deadline.
Describe alternatives you've considered
The current workaround is to save the image to a file accessible by the server and provide its path in the request. This adds unnecessary filesystem I/O and requires the client and server to have access to the same filesystem.
Adding native base64 image support to the server would provide a more interoperable solution while preserving the existing file-path behavior.
Target hardware/use case
Jetson Thor (SM_110, JetPack 7.1, TensorRT 10.13.3.9), using the experimental OpenAI-compatible server to host VLMs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the experimental OpenAI-compatible server's current VLM image-input handling and follow how local file paths enter the existing preprocessing and inference pipeline. Review the linked feat/server_base64_support implementation for the proposed approach, then verify that base64 data URLs work while local file-path support remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100