NVIDIA / NVIDIA/TensorRT-Edge-LLM

feat: Add base64 image support to the OpenAI-compatible server

Open
#191 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

External PR Request
Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Detailed description of the requested feature

The experimental OpenAI-compatible server currently supports VLM image inputs only when the image is provided via a local file path. This prevents clients that use the standard OpenAI vision API format from sending images as base64-encoded data URLs.

I would like to add support for base64-encoded images, following the OpenAI-compatible image_url format, e.g. data:image/jpeg;base64,... or data:image/png;base64,....

The server should decode the base64 image data and pass it through the existing VLM image preprocessing/inference pipeline, while retaining the current file-path support for backwards compatibility.

This would make the experimental server more compatible with existing OpenAI-compatible clients and applications, which commonly transmit images as base64 data URLs rather than exposing local filesystem paths. It would also be useful for deployments where the client and inference server do not share the same filesystem.

My implementation is available at: https://github.com/FABallemand/TensorRT-Edge-LLM/tree/feat/server_base64_support

Per CONTRIBUTING.md, I am opening this issue for approval before requesting code review.

Timeline

No hard deadline.

Describe alternatives you've considered

The current workaround is to save the image to a file accessible by the server and provide its path in the request. This adds unnecessary filesystem I/O and requires the client and server to have access to the same filesystem.

Adding native base64 image support to the server would provide a more interoperable solution while preserving the existing file-path behavior.

Target hardware/use case

Jetson Thor (SM_110, JetPack 7.1, TensorRT 10.13.3.9), using the experimental OpenAI-compatible server to host VLMs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the experimental OpenAI-compatible server's current VLM image-input handling and follow how local file paths enter the existing preprocessing and inference pipeline. Review the linked feat/server_base64_support implementation for the proposed approach, then verify that base64 data URLs work while local file-path support remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.