cloudfoundry / cloudfoundry/bosh-deployment

health monitor keeps triggering scan & fix tasks due to slow nats client

Open
#484 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
138
Forks
234
Avg merge
1d 12h
Merged PRs (30d)
6

Description

Stemcell: bosh-openstack-kvm-ubuntu-jammy-go_agent-raw/1.803
Bosh version: v282.0.0
bosh-openstack-cpi: 55.0.1
Managing 731 deployments, 1273 agents

We ran into the following situation:

The amount of bosh scan and fix tasks keeps being around the count of deployments (500-700 tasks). After the task was done, a new scan and fix has been queued immediately. From metrics perspective the VMs of that director were unresponsive, but when checking with bosh vms or bosh instances, all the VMs were found to be healthy.

In the health_monitor logs the following line appears repetitively:
ERROR : NATS client error: nats: slow consumer, messages dropped

A restart of the health_monitor process helps to unstuck the situation, the bosh scan & fix tasks decrease.
After the restart we are now seeing 1277 Nats onnection, checked with netstat -anp | grep 4222

Before and after the huge queue of scan and fix tasks, the health_monitor logs also show numerous lines like

I, [2025-08-08T06:47:27.906126 #7] INFO : [ALERT] Alert @ 2025-08-08 06:47:27 UTC, severity 1: process is not running
I, [2025-08-08T06:47:27.906275 #7] INFO : (Event logger) notifying director about event: Alert @ 2025-08-08 06:47:27 UTC, severity 1: process is not running

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named. Start with the health_monitor logs, the repeated NATS slow-consumer errors, and the scan and fix task queue behavior before and after restarting health_monitor. Done would require identifying why healthy VMs trigger repeated tasks and defining a verified fix that prevents the queue from continually rebuilding.

Written by the indexing model from the issue text.

Assessment

Domain
infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.