NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

bug: server --auto-reload never picks up config changes on a running server

Open
#2,263 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug status: needs triage
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check docs and existing issues?
  • I have read all the NeMo-Guardrails docs
  • I have updated the package to the latest version before submitting this issue
  • (optional) I have used the develop branch
  • I have searched the existing issues of NeMo-Guardrails
Python version (python --version)

Python 3.14.6

Operating system/version

Linux (container: Debian-based, python:3.14-slim)

NeMo-Guardrails version (if you must use a specific version and not the latest

0.24.0.dev0

Describe the bug

nemoguardrails server --auto-reload is supposed to let an edited config
take effect on a running server with no restart, via a filesystem watchdog
(start_auto_reload_monitoring in nemoguardrails/server/api.py) that
evicts a config's cached LLMRails instance when its files change. In
practice it never works. Two independent root causes, both required to fix
before reload works end-to-end:

  1. Cache-key mismatch: _get_rails() caches instances under
    _generate_cache_key(config_ids, model_name), which appends
    :{model_name} whenever a model is given. GuardrailCheckRequest.model
    is required on /v1/checks, so every real request produces a suffixed
    key (e.g. "regex:gpt-4o"). The watchdog's eviction handler
    (Handler.on_any_event) only checks the bare config directory name
    (e.g. "regex"), which never matches -- eviction is a silent no-op for
    all real traffic. (A second instance of the same mismatch also breaks
    the events-history restore-on-reload path.)

  2. Watchdog silently and permanently stops delivering events on Docker
    bind mounts
    (the common containerized/Kubernetes ConfigMap case):
    Docker's bind-mount inotify forwarding can silently and permanently
    stop -- InotifyBuffer.run() exits its read loop without error but never
    unblocks its downstream queue, so the reader thread hangs with no
    visible symptom.

Steps To Reproduce
  1. Start the server with a config directory containing a named config,
    e.g. --config=/config --auto-reload, where
    /config/myconfig/config.yml exists.
  2. POST /v1/checks with {"model": "some-model", "messages": [...], "guardrails": {"config": "myconfig"}} -- this populates the cache
    under key "myconfig:some-model".
  3. Edit /config/myconfig/config.yml on disk (e.g. change a regex
    pattern, or introduce a YAML syntax error to make the effect
    unambiguous).
  4. Repeat the same POST /v1/checks request as step 2.
Expected Behavior

After editing a mounted config file, the very next request against that
config reflects the new file content (or fails outright, if the edit
introduced a syntax error that a fresh RailsConfig.from_path() would
raise on) -- with no server restart, and this should keep working
indefinitely across repeated edits.

Actual Behavior

Even once eviction correctly matches the right cache key, edits made to a bind-mounted config file are picked up unreliably -- and then silently stop working for the lifetime of the container, with nothing in the logs indicating a problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nemoguardrails/server/api.py at _get_rails(), _generate_cache_key(), start_auto_reload_monitoring(), and Handler.on_any_event(). Reproduce the named /v1/checks workflow, then trace the watchdog InotifyBuffer.run() behavior on a bind mount. Done means repeated edits affect the next request reliably, including edits that introduce a config-loading error, without restarting the server.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.