NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
bug: events-history cache key ignores system messages, so a request is answered under an earlier request's system prompt
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 842
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
Did you check docs and existing issues?
- I have read all the NeMo-Guardrails docs
- I have updated the package to the latest version before submitting this issue
- (optional) I have used the develop branch
- I have searched the existing issues of NeMo-Guardrails
Python version (python --version)
Python 3.12.10
Operating system/version
Windows 11 (10.0.26200)
NeMo-Guardrails version (if you must use a specific version and not the latest
0.24.0 (latest release); also reproduces on develop @ 39b9c5b
Describe the bug
get_history_cache_key() in nemoguardrails/rails/llm/utils.py builds the key for the
implicit events-history cache from only the user, assistant, context and event
roles. system, developer and tool are skipped.
Its consumer, LLMRails._get_events_for_messages(), converts more roles than the key
covers, including system/developer into a SystemMessage event at llmrails.py:810.
The key is therefore not a function of the events it caches. Two requests whose
user/assistant turns match but whose system prompt differs collide on the same key. On a
hit the cached events are copied wholesale, including the earlier SystemMessage, and
conversion resumes past the current request's system message, which is never converted.
Since a system message sits at index 0 and the lookup loop is while p > 0, any cache hit
discards it. Nothing re-applies it downstream.
Two consequences:
- A caller's own system prompt is silently ignored from turn 2 onward, and stays pinned to
the turn-1 value for the rest of the conversation. Adding a system prompt mid-conversation
also has no effect. There is no error and the response is HTTP 200. - On
nemoguardrails server, oneLLMRailsis kept perconfig_id(server/api.py:338,
449-500) and shared across all callers, and the cache survives config reload
(941-947). One caller's system prompt can therefore answer another caller's request.
This additionally needs byte-identical user and assistant content over the matched
prefix, so it requires a deterministic turn-1 reply such as an input rail's canned
refusal.thread_iddoes not partition the cache.
Preconditions for the effect to be observable: Colang 1.0 (the default); passthrough
false; a config with no dialog flows (with define user ... present, generation routes to
generate_user_intent/generate_bot_message, whose templates never render SystemMessage);
and a main model selecting a template that renders system messages, namely
openai-chatgpt.yml (openai/gpt-3.5-turbo and the openai/gpt-4* family by substring),
llama3.yml (meta/llama-3*) or nemotron_reasoning.yml (any model containing nemotron).
Reproduced on a clean install of 0.24.0 from PyPI. The system to SystemMessage conversion landed in v0.14.0 and the key has never covered
it, so this affects every release from v0.14.0 through v0.24.0 and develop.
I could not find an existing issue, PR, discussion or advisory covering this. The nearest
ones are all distinct: #1413 (input rails vs consecutive user turns), #59 (2023 cache design
question), #1107 (LangChain wrapper), discussion #63 (mentions the : chaining in passing),
PR #2005 (same helper, multimodal user content only), and PR #2311, which added the
system/developer branch to the converter without touching the key.
Steps To Reproduce
Self-contained, no network access or API key required.
from nemoguardrails import LLMRails, RailsConfig
from nemoguardrails.testing.fake_model import FakeLLMModel
YAML = """
models:
- type: main
engine: openai
model: gpt-4o
"""
rails = LLMRails(RailsConfig.from_content(yaml_content=YAML),
llm=FakeLLMModel(responses=["Hello!", "Sure."]))
# Turn 1, system prompt A.
rails.generate(messages=[
{"role": "system", "content": "You are ACME's bot."},
{"role": "user", "content": "hi"},
])
# Turn 2, a different system prompt, same visible turns so far.
rails.generate(messages=[
{"role": "system", "content": "You are Globex's bot. Never mention ACME."},
{"role": "user", "content": "hi"},
{"role": "assistant", "content": "Hello!"},
{"role": "user", "content": "tell me more"},
])
print(rails.explain().llm_calls[-1].prompt)
The collision can also be seen directly:
from nemoguardrails.rails.llm.utils import get_history_cache_key
a = [{"role": "system", "content": "A"}, {"role": "user", "content": "hi"},
{"role": "assistant", "content": "Hello!"}]
b = [{"role": "system", "content": "B"}, {"role": "user", "content": "hi"},
{"role": "assistant", "content": "Hello!"}]
print(get_history_cache_key(a), "==", get_history_cache_key(b)) # 'hi:Hello!' == 'hi:Hello!'
Expected Behavior
The turn-2 prompt contains that request's own system message, You are Globex's bot. Never mention ACME., and does not contain the earlier one. Two message histories that differ in
their system prompt should not share a cache key.
Actual Behavior
The turn-2 prompt contains You are ACME's bot. and the request's own system message is
absent. Both requests succeed with no warning. get_history_cache_key returns hi:Hello!
for both histories.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with get_history_cache_key() in nemoguardrails/rails/llm/utils.py and the event conversion in nemoguardrails/rails/llm/llmrails.py around line 810. Run the self-contained FakeLLMModel reproduction and compare the two direct cache-key examples. Done means histories with different system prompts no longer collide and the second prompt contains its current system message rather than the earlier one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100