nsd: updating a job with service in nomad provider causes agent error logs
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
While debugging #16238 I noticed just updating a job with a service in the nomad provider can trigger agent Error logs, e.g.
```
2023-03-01T08:57:24.160-0600 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
2023-03-01T08:57:24.160-0600 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=127.0.0.1:4647
2023-03-01T08:57:24.160-0600 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=127.0.0.1:4647
2023-03-01T08:57:24.160-0600 [INFO] client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-ef1dd8fa-b15c-4d9e-3fc4-bede2397bbea-group-group-test-http namespace=default
```
Unclear if these are benign (the end result is OK - the expected service is in the registry).
To update just change `meta.version`.
```hcl
job "bug" {
group "group" {
restart {
interval = "3m"
attempts = 3
delay = "15s"
mode = "delay"
}
network {
port "http" { to = 8080 }
}
meta {
version = 2
}
service {
name = "test"
port = "http"
address_mode = "host"
tags = []
provider = "nomad"
check {
type = "http"
port = "http"
path = "/"
interval = "12s"
timeout = "6s"
check_restart {
limit = 3
grace = "10s"
}
}
}
task "task" {
driver = "docker"
config {
image = "python:3"
command = "python3"
args = ["-m", "http.server", "${NOMAD_PORT_http}"]
ports = ["http"]
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue using the HCL job shown in the report, changing only meta.version while a Nomad service is registered. Start by tracing the service-registration update and deletion behavior associated with the reported RPC and agent logs; determine whether the errors are benign, then verify that updating the job preserves the expected service without erroneous error logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100