NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
feature: public `api_key` property for in-place LLM credential rotation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 842
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
Did you check the docs?
- I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.
Applications that build one long-lived LLMRails/OpenAIChatModel instance at process startup (to avoid per-request construction overhead) and authenticate against their LLM provider with a short-lived, rotating bearer token (OAuth client-credentials, AWS STS-style tokens, internal gateway tokens, etc.) have no supported way to update that credential in place.
Today the only two options are:
- Reach into private attributes two levels deep (
rails.llm._client._api_key), which is unsupported and can silently break on any minor/patch upgrade with no type or import error — just silent 401s from the provider. - Rebuild the model via
DefaultFramework.create_model()and swap it in withLLMRails.update_llm(). This looks like the supported path, butDefaultFramework._clientscachesOpenAICompatibleClientinstances keyed by(base_url, api_key, ...)and never evicts entries, so every token rotation leaks a newhttpx.AsyncClientconnection pool for the life of the process.
Short-lived, periodically-rotated bearer tokens are a common integration pattern for enterprise LLM gateways, not something specific to one deployment, so this gap affects any application holding a long-lived rails/model instance.
Describe the solution you'd like
Add a small, purely additive api_key property (with setter) so callers have
a public, documented way to rotate the credential on an already-built model:
BaseClient.api_key(nemoguardrails/llm/clients/base.py): getter/setter over the existing_api_keyattribute. Safe to call between requests, since_build_headers()already re-reads_api_keyfresh on every call.OpenAIChatModel.api_key(nemoguardrails/llm/models/openai_chat.py): delegates to the underlying client'sapi_key.InstrumentedLLMModel.api_key(nemoguardrails/llm/models/instrumented.py):LLMRailswraps the model in this decorator whenever tracing or metrics are enabled, so the tracing/metrics wrapper needs to forwardapi_keytoo, otherwise assignment silently creates a stray attribute on the decorator and never reaches the real client.- Document on the
LLMModelprotocol (nemoguardrails/types.py) thatapi_keyis a bearer-token-specific extension, not a required protocol member, so callers know tohasattr-check before assigning.
With this, credential rotation becomes:
if hasattr(rails.llm, "api_key"):
rails.llm.api_key = new_token
No existing method signature, default, or behavior changes; this only adds new
public surface.
Describe alternatives you've considered
- Keep using the private-attribute workaround (
rails.llm._client._api_key) - rejected, unsupported and can silently break across versions. - Rebuild the model and call
LLMRails.update_llm()on every rotation - rejected, leaks anOpenAICompatibleClient+httpx.AsyncClientconnection pool per rotation viaDefaultFramework._clients, since that cache is keyed byapi_keyand entries are never evicted. - Bound/evict
DefaultFramework._clientsinstead of adding a setter - would fix the leak but not the missing-public-accessor gap, and is a larger, separate change to shared framework caching behavior.
Additional context
- A draft implementation (property + setter across
BaseClient,OpenAIChatModel,InstrumentedLLMModel, plus theLLMModeldocstring note) and unit tests already exist on a local branch, ready to open as a PR once this issue is triaged and assigned. - The
DefaultFramework._clientsunbounded-cache issue is a related but distinct bug; flagging it here for awareness but proposing it as a separate follow-up rather than folding it into this change.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with nemoguardrails/llm/clients/base.py, nemoguardrails/llm/models/openai_chat.py, nemoguardrails/llm/models/instrumented.py, and the LLMModel protocol in nemoguardrails/types.py. Review the existing unit tests on the draft branch and run them before and after the change. Done means callers can rotate a bearer token through the public model property, including when the model is instrumented, without changing existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- authentication, backend-api-design
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100