ModelEngine-Group / ModelEngine-Group/nexent
[Bug] `MonitoringRecordBuffer._flush_loop` sleeps an extra `flush_interval` after stop signal
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 731
- Avg merge
- 19h 34m
- Merged PRs (30d)
- 172
Description
sdk/nexent/monitor/monitoring.py:1081-1099:
def _flush_loop(self) -> None:
while self._running:
try:
now = time.time()
...
except Exception as e:
logger.error(f"Error in monitoring flush loop: {e}")
for _ in range(10):
if not self._running:
return
time.sleep(self._flush_interval / 10)
Two issues:
-
Latency after exception. If
_flush_to_dbraises, the loop logs and falls through into the inner sleep. There is no fast-exit on error, so a transient DB outage means a fullflush_intervallag on every error, even if the underlying problem cleared within a second. With the default 30 s flush interval that's 30 s of backed-up buffer. -
Shutdown can hang up to
flush_interval / 10seconds. During the currently-activetime.sleepslice, theif not self._runningcheck has already passed. Withflush_interval = 30(default), that's a 3 s shutdown delay. Setflush_interval = 300(5 min) for low-volume tenants and shutdown blocks 30 s.
A threading.Event solves both:
def __init__(self):
...
self._stop_event = threading.Event()
def _flush_loop(self):
while not self._stop_event.is_set():
try:
...
except Exception as e:
logger.error(...)
self._stop_event.wait(timeout=self._flush_interval)
def stop(self):
self._stop_event.set()
Event.wait(timeout=...) returns immediately when set() is called from another thread.
Category: A/H. Severity: Low.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in sdk/nexent/monitor/monitoring.py:1081-1099 at MonitoringRecordBuffer._flush_loop, then inspect its init and stop() methods. Trace both the exception path and shutdown path, and verify that the stop signal interrupts waiting immediately while errors do not add an unnecessary full interval delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100