cloudfoundry / cloudfoundry/cloud_controller_ng

Allow failure threshold for application health check

Open
#3,435 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

unscheduled
Dominant language
Ruby
Stars
207
Forks
373
Avg merge
2d 12h
Merged PRs (30d)
56

Description

Issue

We observed a lot of application crashes due to health check (with http request) timeout, but all other http requests were actually working right before crashes, and also all other metrics were good.

Actually, our health check endpoint is quite fast without any other logic, and also we increased the timeout to 20 seconds but it doesn't help too much.

Expected result

We are not sure why some of health checks fail due to timeout, might be CPU throttling.
But we expect that the runtime gives the application another chance to do another health check.

Current result

Application instance would be restarted after only one single health check failure.

Possible Fix

Adding failure threshold for health check, and after health check fails failureThreshold times in a row, the runtime considers that the overall check has failed and the container is not healthy/live.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name a file, test, or entry point. Begin by locating the application health-check timeout and the logic that restarts an instance after one failure, then inspect related configuration and tests. Done means consecutive failures are counted and an instance is considered unhealthy only after the configured threshold is reached.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.