NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

feature: public `api_key` property for in-place LLM credential rotation

Open
#2,299 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement status: needs triage
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check the docs?
  • I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.

Applications that build one long-lived LLMRails/OpenAIChatModel instance at process startup (to avoid per-request construction overhead) and authenticate against their LLM provider with a short-lived, rotating bearer token (OAuth client-credentials, AWS STS-style tokens, internal gateway tokens, etc.) have no supported way to update that credential in place.

Today the only two options are:

  • Reach into private attributes two levels deep (rails.llm._client._api_key), which is unsupported and can silently break on any minor/patch upgrade with no type or import error — just silent 401s from the provider.
  • Rebuild the model via DefaultFramework.create_model() and swap it in with LLMRails.update_llm(). This looks like the supported path, but DefaultFramework._clients caches OpenAICompatibleClient instances keyed by (base_url, api_key, ...) and never evicts entries, so every token rotation leaks a new httpx.AsyncClient connection pool for the life of the process.

Short-lived, periodically-rotated bearer tokens are a common integration pattern for enterprise LLM gateways, not something specific to one deployment, so this gap affects any application holding a long-lived rails/model instance.

Describe the solution you'd like

Add a small, purely additive api_key property (with setter) so callers have
a public, documented way to rotate the credential on an already-built model:

  • BaseClient.api_key (nemoguardrails/llm/clients/base.py): getter/setter over the existing _api_key attribute. Safe to call between requests, since _build_headers() already re-reads _api_key fresh on every call.
  • OpenAIChatModel.api_key (nemoguardrails/llm/models/openai_chat.py): delegates to the underlying client's api_key.
  • InstrumentedLLMModel.api_key (nemoguardrails/llm/models/instrumented.py): LLMRails wraps the model in this decorator whenever tracing or metrics are enabled, so the tracing/metrics wrapper needs to forward api_key too, otherwise assignment silently creates a stray attribute on the decorator and never reaches the real client.
  • Document on the LLMModel protocol (nemoguardrails/types.py) that api_key is a bearer-token-specific extension, not a required protocol member, so callers know to hasattr-check before assigning.

With this, credential rotation becomes:

if hasattr(rails.llm, "api_key"):
    rails.llm.api_key = new_token

No existing method signature, default, or behavior changes; this only adds new
public surface.

Describe alternatives you've considered
  • Keep using the private-attribute workaround (rails.llm._client._api_key) - rejected, unsupported and can silently break across versions.
  • Rebuild the model and call LLMRails.update_llm() on every rotation - rejected, leaks an OpenAICompatibleClient + httpx.AsyncClient connection pool per rotation via DefaultFramework._clients, since that cache is keyed by api_key and entries are never evicted.
  • Bound/evict DefaultFramework._clients instead of adding a setter - would fix the leak but not the missing-public-accessor gap, and is a larger, separate change to shared framework caching behavior.
Additional context
  • A draft implementation (property + setter across BaseClient, OpenAIChatModel, InstrumentedLLMModel, plus the LLMModel docstring note) and unit tests already exist on a local branch, ready to open as a PR once this issue is triaged and assigned.
  • The DefaultFramework._clients unbounded-cache issue is a related but distinct bug; flagging it here for awareness but proposing it as a separate follow-up rather than folding it into this change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nemoguardrails/llm/clients/base.py, nemoguardrails/llm/models/openai_chat.py, nemoguardrails/llm/models/instrumented.py, and the LLMModel protocol in nemoguardrails/types.py. Review the existing unit tests on the draft branch and run them before and after the change. Done means callers can rotate a bearer token through the public model property, including when the model is instrumented, without changing existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
authentication, backend-api-design
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.