ModelEngine-Group / ModelEngine-Group/nexent

[Bug] `MonitoringRecordBuffer._flush_loop` sleeps an extra `flush_interval` after stop signal

Open
#3,403 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.9k
Forks
731
Avg merge
19h 34m
Merged PRs (30d)
172

Description

sdk/nexent/monitor/monitoring.py:1081-1099:

def _flush_loop(self) -> None:
    while self._running:
        try:
            now = time.time()
            ...
        except Exception as e:
            logger.error(f"Error in monitoring flush loop: {e}")

        for _ in range(10):
            if not self._running:
                return
            time.sleep(self._flush_interval / 10)

Two issues:

  1. Latency after exception. If _flush_to_db raises, the loop logs and falls through into the inner sleep. There is no fast-exit on error, so a transient DB outage means a full flush_interval lag on every error, even if the underlying problem cleared within a second. With the default 30 s flush interval that's 30 s of backed-up buffer.

  2. Shutdown can hang up to flush_interval / 10 seconds. During the currently-active time.sleep slice, the if not self._running check has already passed. With flush_interval = 30 (default), that's a 3 s shutdown delay. Set flush_interval = 300 (5 min) for low-volume tenants and shutdown blocks 30 s.

A threading.Event solves both:

def __init__(self):
    ...
    self._stop_event = threading.Event()

def _flush_loop(self):
    while not self._stop_event.is_set():
        try:
            ...
        except Exception as e:
            logger.error(...)
        self._stop_event.wait(timeout=self._flush_interval)

def stop(self):
    self._stop_event.set()

Event.wait(timeout=...) returns immediately when set() is called from another thread.

Category: A/H. Severity: Low.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sdk/nexent/monitor/monitoring.py:1081-1099 at MonitoringRecordBuffer._flush_loop, then inspect its init and stop() methods. Trace both the exception path and shutdown path, and verify that the stop signal interrupts waiting immediately while errors do not add an unnecessary full interval delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.