Experiencing intermittent high cpu spikes on a Nomad job.

Nomad version

Nomad v1.5.6
BuildDate 2023-05-19T18:26:13Z
Revision 8af70885c02ab921dedbdf6bc406a1e886866f80

Operating system and Environment details

Linux 4.14.309-231.529.amzn2.x86_64 GNU/Linux

Issue

We have a nomad job that is acting as a service checker, it has a dummy docker image and runs a check on the service block, it also has an upstream to another service on port 18005. This check is a custom script written by us and runs every 60s. The script could be simply explained as a curl, which uses localhost:18005 as an argument.

The problem is that we are seeing high cpu spikes, well above what they should be. They are probably related to the check script because the spikes always occur in different minutes of the same second and because we tested the job without the check block and it was working fine.

Reproduction steps

1. Create a check script, that executes a curl and returns 0 or 1 depending on if its successfull, to use with the job below.
2. Run the job file provided below on a Nomad datacenter with Consul integration.

Expected Result

Metrics percentage way below a 100% because is a simple curl script

Actual Result

It’s also weird that some spikes are 100% some are 90% and some are 0% … it is not consistent.

Job file

job "service-checker" {
  datacenters = ["dc1"]
  namespace   = "default"

  constraint {
    attribute = "${node.class}"
    value     = "default"
  }

  group "service-checker" {
    count = "1"

    update {
      stagger      = "1m"
      max_parallel = 1
    }

    restart {
      interval = "2m"
      attempts = 5
      delay    = "15s"
      mode     = "delay"
    }

    network {
      mode = "bridge"
    }

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    task "service-checker" {
      driver = "docker"

      config {
        image = "dummydockerimage-servicechecker:v0.0.1"
        command = "tail"
        args    = ["-f", "/dev/null"]
      }

      resources {
        cpu    = 500
        memory = 200
      }

      template {
        data        = "{{key \"scripts/service-checker/check_status_ok\"}}"
        destination = "local/check_status_ok.sh"
        perms       = "555"
        env         = false
      }
    }

    # Services_checker
    service {
      name = "service-checker-envoy"
      tags = ["sample-tags enabled=true"]
      check {
        name     = "envoy-script-check"
        task     = "service-checker"
        type     = "script"
        command  = "/local/check_status_ok.sh"
        args     = ["http://localhost:18005/clusters"]
        interval = "60s"
        timeout  = "5s"
      }
      connect {
        sidecar_service {
          tags = ["connect-proxy"]
          proxy {
            upstreams {
              destination_name = "envoy-status"
              local_bind_port  = 18005
            }
          }
        }
      }
    }
  }
}

Nomad Client logs

This is the error we see in the logs of the node that executed this job

2023-10-03T15:14:39.341Z [DEBUG] client.driver_mgr.exec.executor.nomad: time="2023-10-03T15:14:39Z" level=warning msg="cannot serialize hook of type configs.FuncHook, skipping": alloc_id=7896e801-5219-e023-c23121b-83d51432d079e driver=exec task_name=service-checker

Start by reproducing the provided Nomad job file with the script check enabled, then compare it with the version without the check block. Inspect the Nomad client log warning about serializing configs.FuncHook and correlate it with the reported CPU spikes. Done means identifying a concrete cause and documenting or fixing the behavior with evidence from the reproduction.