NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
feature: Allow the specification of request-time model API keys via request headers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 842
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
Did you check the docs?
- I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.
Right now, there's no way that I can see to pass request-time API keys to the NeMo Guardrails server. That means the deployer of the server must always use their own API key to authenticate to any downstream LLMs, instead of the caller's API key. This makes user usage tracking on the downstream LLMs very difficult.
Describe the solution you'd like
Some mechanism to pass user API keys through the NeMo Guardrails server to the downstream LLMs, at least for /v1/chat/completions models.
Ideally, this would allow the specification of an auth header in the inbound request to the server that gets translated into an "Authorization: Bearer xyz header when making the LLM server request, e.g.:
models:
- type: main
engine: openai
model: gpt-4
api_key_header: "X-API-User-Key" # The name of the HTTP header containing the API key
parameters:
openai_api_key: 'deployer-key'
Then, any inbound request:
- with a header
X-API-User-Key: $somethingwill sendAuthorization: Bearer $something - without a matching header will send
Authorization: Bearer $deployer-key
This provides a a number of benefits versus recycling the inbound Authorization header:
- We don't need to enforce that the authentication key to the NeMo Server matches the authentication key to the model
- We can set
openai_api_keyto something likeinvalid_tokento enforce that only authenticated users can access the model, without having to provide unfettered access via the deployer's token.
Describe alternatives you've considered
- Redeploying an instance of the server for each new user token <- this is obviously a bad approach
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the server's /v1/chat/completions request handling and trace how the YAML models configuration reaches downstream LLM requests. Define the header-to-Authorization fallback behavior described in the issue, including requests without the configured header, and verify that the deployer key remains the fallback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, authentication, backend, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100