hashicorp / hashicorp/nomad

nsd: updating a job with service in nomad provider causes agent error logs

Open
#16,286 0 comments 1 reaction 0 assignees View on GitHub
stage/needs-investigation theme/service-discovery/nomad type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While debugging #16238 I noticed just updating a job with a service in the nomad provider can trigger agent Error logs, e.g.

```
2023-03-01T08:57:24.160-0600 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
2023-03-01T08:57:24.160-0600 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=127.0.0.1:4647
2023-03-01T08:57:24.160-0600 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=127.0.0.1:4647
2023-03-01T08:57:24.160-0600 [INFO] client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-ef1dd8fa-b15c-4d9e-3fc4-bede2397bbea-group-group-test-http namespace=default
```

Unclear if these are benign (the end result is OK - the expected service is in the registry).

To update just change `meta.version`.

```hcl
job "bug" {
group "group" {

restart {
interval = "3m"
attempts = 3
delay = "15s"
mode = "delay"
}

network {
port "http" { to = 8080 }
}

meta {
version = 2
}

service {
name = "test"
port = "http"
address_mode = "host"
tags = []
provider = "nomad"
check {
type = "http"
port = "http"
path = "/"
interval = "12s"
timeout = "6s"

check_restart {
limit = 3
grace = "10s"
}
}
}

task "task" {
driver = "docker"
config {
image = "python:3"
command = "python3"
args = ["-m", "http.server", "${NOMAD_PORT_http}"]
ports = ["http"]
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue using the HCL job shown in the report, changing only meta.version while a Nomad service is registered. Start by tracing the service-registration update and deletion behavior associated with the reported RPC and agent logs; determine whether the errors are benign, then verify that updating the job preserves the expected service without erroneous error logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.