vllm-project / vllm-project/aibrix

Support per user api-key for multi-tenant use case

Open
#753 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

## Background

Currently, vLLM only supports a single API key for authentication, making it difficult to share the inference engine across multiple tenants. Extending vLLM to support multiple keys is an option, but this would be a static solution. A more flexible approach is needed to handle multi-tenant API key management dynamically.

## Proposed Solutions

### Option 1: Extend vLLM to Support External Authentication

- vLLM integrates with an external authentication server to validate API keys dynamically.
- This approach allows for greater flexibility but introduces external dependencies. overhead is another concern

### Option 2: Manage API Keys Outside of vLLM

#### Option 2a: User-Managed Authentication (Bring Your Own Stack)
- Users adopt an external authentication solution (e.g., Istio, OAuth, or API gateways) to manage API keys.

#### Option 2b: Extend AIBrix Gateway for Multi-Tenant API Key Management
- AIBrix Gateway already has a basic user concept and rate-limiting control.
- The extension would associate users with API keys, providing built-in multi-tenancy support.

## Future Considerations

In addition to authentication, we want to support tenant-aware optimizations within vLLM. The gateway should attach tenant metadata (e.g., X-Tenant-ID, X-Priority, JWT claims) before forwarding the request to vLLM. This would enable the inference engine to make tenant-aware optimizations, such as priority-based scheduling or resource allocation.

Open Questions
- Which approach aligns best with the vLLM architecture?
- Should vLLM natively support dynamic authentication, or should this be handled externally?
- How can we ensure a smooth integration between vLLM and the authentication layer without introducing significant overhead?

/cc @simon-mo @robertgshaw2-redhat @gaocegege @kerthcet

### Use Case

Support multi-tenancy for vLLM

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.