spring-projects / spring-projects/spring-boot
Actuator - support optional downstream services in health
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 81.5k
- Forks
- 42.7k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 65
Description
I'm using the Actuator project to display health statuses for downstream services. My application is utilizing an optional downstream service; the service is used all the time normally but we have fallbacks in place if it goes down for whatever reason.
Optional services are not supported at this time and I'd like to request that we change that. The problem is that the HealthIndicator for the optional service will show the downstream service as being down, that DOWN status will be aggregated to the overall service status, and the health check will fail because of a non-200 status code. This is not desirable. Optional downstream services should show up as "DOWN" but not have that status aggregated.
This enhancement has been requested in the past a couple of times over the years:
- https://github.com/spring-projects/spring-boot/issues/10550
- https://github.com/spring-projects/spring-boot/issues/18701
The general recommendation was to build a custom HealthAggregator. HealthAggregator has been deprecated so the replacement recommendation is a StatusAggregator + Health Groups. This does not work.
Here's a StatusAggregator that I built for a health group named "optional":
@Qualifier("optional")
@Component
public class OptionalStatusAggregator implements StatusAggregator {
@Override
public Status getAggregateStatus(Set<Status> statuses) {
return Status.UP;
}
}
And here's my actuator configuration:
management:
endpoint:
health:
show-details: always
group:
optional:
include:
- someservice
endpoints:
web:
base-path: /
exposure:
include:
- info
- health
The intent is to always show "UP" for the optional health group since none of these are critical for the overall system health. The components in the group will still have their individual statuses but they are to be ignored once aggregated.
If you hit the health group's health endpoint (localhost:8080/health/optional), the aggregation works as intended. However, my custom status aggregator is not used for the overall health endpoint (localhost:8080/health) so the "DOWN" status is propagated up to the service health.
The only solution I can see is to create a StatusAggregator that's used for everything. However, the StatusAggregator does not provide where the statuses came from, only the raw Status values of "UP", "DOWN", etc. in the form of a Set. There is no way to have the aggregator ignore the statuses of some downstream services because there's no context. It was proposed that this be fixed in https://github.com/spring-projects/spring-boot/issues/23759 but the team decided not to extend the API of the StatusAggregator.
The recommendation was to instead create a custom status called "PARTIAL_DOWN" and create a status aggregator to handle that. The problem with this recommendation is that the downstream service that is actually down will show up with the status of "PARTIAL_DOWN" which is not true and the overall status will be "PARTIAL_DOWN" (kind of true) or "UP" (true) depending on how the StatusAggregator is implemented.
I'd like you to reconsider adding the component context information into StatusAggregator as an enhancement. This information is needed in order to do any sort of meaningful status aggregation. In the current state, the existing SimpleStatusAggregator is effectively the only possible way to aggregate statuses unless you choose to hardcode the status like I did in my example (which is not useful for the overall health endpoint status aggregation). At this time, I will not be able to add my optional downstream services to the health endpoint and there will unfortunately not be any visibility into how they are doing from the application's perspective.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Actuator health endpoint, StatusAggregator, and Health Groups described in the issue, and trace how the overall health status is aggregated. Done should preserve individual optional component statuses while preventing their DOWN status from affecting the overall health result, with coverage for both the optional group and overall endpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring-boot
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100