spring-cloud / spring-cloud/spring-cloud-consul

Expected behaviour with Consul Service Discovery and queryPassing=true

Open
#771 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
822
Forks
539
Avg merge
9h 31m
Merged PRs (30d)
7

Description

When using a basic configuration to support consul service discovery, we're seeing some inconsistent behaviour when using the queryPassing setting. An example configuration is shown below.

spring:
    cloud:
        consul:
            config:
                failFast: false
            enabled: true
            host: localhost
            port: 8500
            discovery:
                register: false
                enabled: true
                registerHealthCheck: false
                instanceId: ${spring.application.name}:${random.value}
                query-passing: true       
        gateway:     
            discovery:
                locator:
                    enabled: true
                    predicates:
                    - name: Path
                      args:
                        pattern: "'/services/' + serviceId + '/**'"
                    filters:
                    -   name: RewritePath
                        args:
                            regexp: "'/services/' + serviceId + '/(?<remaining>.*)'"
                            replacement: "'/${remaining}'"

When queryPassing is false, we observe:

  • Service starts and registers with consul - endpoints not yet started.
  • Route is registered in gateway as expected.
  • Service health check goes to passing.
  • If multiple instances are started, the load balancer will occasionally choose one that is not passing healthchecks.

When queryPassing is true and the service is up before gateway, we observe:

  • Gateway starts, detects new service and registers route.
  • If multiple instances are running, load balancer will only choose ones with healthchecks passing.

When queryPassing is true and the service is started after gateway, we observe:

  • Service starts and registers with consul - endpoints not yet started.
  • Route is not registered with gateway because /actuator/health not yet up on the service.
  • Service health check goes to passing.
  • Service route returns 404.
  • Another service starts and registers with consul, same thing happens, but first service now has its route registered because healthcheck is passing.

My initial thoughts on this are that we should be doing one of the following:

  • Don't include queryPassing when establishing routes and only use it for the load balancer.
  • Add some logic to the Consul catalog scan so that it will refresh routes when a service that was failing goes to passing. I can't see a way of doing this without making significantly more requests to Consul.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied Consul, gateway, and query-passing configurations, comparing behavior when the service starts before versus after the gateway. Read the Consul catalog scan and load-balancer integration points mentioned in the report, then determine how route registration reacts to health-check state. Done means the chosen behavior is documented and service routes and load balancing remain consistent for both startup orders.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.