NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

feature: Allow the specification of request-time model API keys via request headers

Open
#1,676 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement status: needs triage
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check the docs?
  • I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.

Right now, there's no way that I can see to pass request-time API keys to the NeMo Guardrails server. That means the deployer of the server must always use their own API key to authenticate to any downstream LLMs, instead of the caller's API key. This makes user usage tracking on the downstream LLMs very difficult.

Describe the solution you'd like

Some mechanism to pass user API keys through the NeMo Guardrails server to the downstream LLMs, at least for /v1/chat/completions models.

Ideally, this would allow the specification of an auth header in the inbound request to the server that gets translated into an "Authorization: Bearer xyz header when making the LLM server request, e.g.:

models:
  - type: main
    engine: openai
    model: gpt-4
    api_key_header: "X-API-User-Key"  # The name of the HTTP header containing the API key
    parameters:
        openai_api_key: 'deployer-key'

Then, any inbound request:

  • with a header X-API-User-Key: $something will send Authorization: Bearer $something
  • without a matching header will send Authorization: Bearer $deployer-key

This provides a a number of benefits versus recycling the inbound Authorization header:

  1. We don't need to enforce that the authentication key to the NeMo Server matches the authentication key to the model
  2. We can set openai_api_key to something like invalid_token to enforce that only authenticated users can access the model, without having to provide unfettered access via the deployer's token.
Describe alternatives you've considered
  1. Redeploying an instance of the server for each new user token <- this is obviously a bad approach
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the server's /v1/chat/completions request handling and trace how the YAML models configuration reaches downstream LLM requests. Define the header-to-Authorization fallback behavior described in the issue, including requests without the configured header, and verify that the deployer key remains the fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, authentication, backend, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.