OpenAI Proxy

Nobody has claimed this yet.

Hello all! Was curious if anyone has used FastChat as a proxy around OpenAI/Azure endpoints.

Use case:

We have multiple nodes all making calls to either Azure or OpenAI chat endpoints. We have two main problems.

* We'd like to store all prompts and completions made to OpenAI in a central repository for our data team. While this can be achieved using Langchain callbacks, it's not the best solution.
* We throttle our completions to stay within our (elevated) rate limits. We've considered using a "leaky bucket" approach with redis to handle rate limiting, or having this centralized proxy server act as the global rate limit , which can simply queue requests during peak times and otherwise let requests through instantaneously, which, in aggregate, would be much faster than our hard-coded token throughput limits for each of our processes that is making API calls.

If anyone has done this before with FastChat, or used something similar, I'd love to hear about it. Cheers!

The issue names no files, tests, or entry points. First clarify whether FastChat should support an OpenAI/Azure proxy, centralized prompt and completion storage, and global request throttling; the report does not define completion criteria.

Written by the indexing model from the issue text.