feat: Offline batched inference based on OpenAI offline batching API

This issue is created to track the conversation about the offline batch API.

gaocegege@

I was just wondering if there are any plans to add openai api compatible offline batch support in the router. I saw a comment about it https://github.com/vllm-project/vllm/issues/1636#issuecomment-1816831493, and it looks like this feature needs file interfaces for uploads, which might not fit well with vLLM. It could be a nice addition to the router, acting as a bridge between users and vLLM.

ApostaC@

Seems like there should be multiple design choices to handle file uploads. Will add this to the roadmap (which will be released as an issue in the project tomorrow). Maybe we can have more detailed discussion there?

simon-mo@

I also agree this can be a lightweight optional component in the stack! given in K8s you can provision persistent volume easily or mount s3-fuse

No files, tests, or entry points are named. Start by reviewing the router and the linked OpenAI offline batch API documentation, then examine the project's roadmap discussion for the intended design around file uploads and persistent storage. Done means an agreed implementation plan and supported offline batching behavior, but the issue does not define the required interface.