Job remains in FAILED state with no allocations and no further recovery attempts

### Nomad version
2.0.0

### Operating system and Environment details
Ubuntu 22.04

### Issue
We occasionally observe a situation where a Nomad job ends up in the FAILED state with no running allocations, and Nomad no longer attempts to start a new allocation.

From what we understand, the job does not enter the normal restart or reschedule flow. Instead, it appears to be handled by the update deployment logic.

Our assumption is the following:

a node goes down, causing the allocation to be replaced (either on another node or on the same node after it comes back);
since this replacement somehow becomes a new deployment, the allocation is treated as a canary (canary = 1);
if the canary allocation fails during startup, it is marked as FAILED;
after that, Nomad does not create any additional allocations, even though there is no healthy allocation left for the job.

As a result, the job remains permanently in the FAILED state with zero running allocations.

Is this expected behavior, or is this a bug?

It feels unexpected that a service job can end up in a state where there are no running allocations and no further recovery attempts are made.
E.g. In K8S controller always tries to run/place containers inside pods.

The only way we have found to recover is to either:
manually stop and start the job from the UI, or redeploy again through CI.

I've attached the complete job specification below.

#### Expected Result
Job infinitely restarts its allocations on failures trying to run successfully and pass all health checks. No manual intervention needed.

#### Actual Result
Job stucks in **FAILED** state forever.
BUT If we increase healthy_deadline and progress_deadline - it goes in **DEPLOYING** state for the deadlines time. But no further allocations trying to run, and we see deployment status `Deployment is running pending automatic promotion`.

### Job file (if appropriate)
```hcl
job "${APP_NAME}" {
datacenters = ["dc1"]
type = "service"
namespace = var.nomad_namespace

update {
healthy_deadline = "5m"
progress_deadline = "10m"
auto_promote = true
canary = 1
min_healthy_time = "10s"
}

reschedule {
unlimited = true
delay = "5s"
delay_function = "constant"
}

vault {
policies = ["credentials"]
}

meta {
table = "${APP_NAME}"
uuid = uuidv4()
}

group "${APP_NAME}" {

disconnect {
lost_after = "24h"
replace = true
reconcile = "keep_replacement"
}

network {
mode = "cni/nomad-custom"

port "api" {
to = var.api_port
}
}

service {
name = "${APP_NAME}"
port = "api"

check {
port = "api"
type = "http"
path = "/status"
interval = "20s"
timeout = "5s"

check_restart {
limit = 10
grace = "5s"
ignore_warnings = false
}
}

meta {
auto_metrics = "enabled"
auto_metrics_path = "/metrics"
namespace = "${var.nomad_namespace}"
}
}

task "INITIALIZER" {

lifecycle {
hook = "prestart"
}

restart {
attempts = 0
delay = "5s"
interval = "200s"
mode = "fail"
}

driver = "docker"

config {
image = "${INITIALIZER_IMAGE}"
command = "bash"
args = ["-c", "python3 /root/main.py"]
volumes = ["/srv:/srv"]
force_pull = true
}

template {
destination = "secrets/file.env"
change_mode = "restart"
env = true
data = "NOMAD_TOKEN={{ with secret \"kv2/data/common/token\" }}{{- .Data.data.token -}}{{ end }}"
}
}

task "${APP_NAME}" {

logs {
max_files = 2
max_file_size = 10
}

restart {
attempts = 1
delay = "5s"
interval = "20s"
mode = "delay"
}

driver = "docker"

resources {
cpu = "${CPU_REQ}"
memory = "${MEM_REQ}"
}

config {
security_opt = [
"apparmor=dbus-avahi-docker",
]

image = "${REGISTRY_IMAGE}:${IMAGE_TAG}"
image_pull_timeout = "20m"

ports = ["api"]

volumes = [
"local/config.json:/app/build/config.json",
"/srv:/srv",
"/var/run/dbus:/var/run/dbus"
]

devices = var.inputType == "ndi" ? [
{ host_path = "/dev/nvme0n1", container_path = "/dev/nvme0n1" },
{ host_path = "/dev/nvme1n1", container_path = "/dev/nvme1n1" },
{ host_path = "/dev/nvme2n1", container_path = "/dev/nvme2n1" },
{ host_path = "/dev/nvme3n1", container_path = "/dev/nvme3n1" }
] : [
{ host_path = "/dev/blackmagic", container_path = "/dev/blackmagic" }
]

ipc_mode = "host"
}

template {
source = "${NOMAD_ALLOC_VAR_FILE_PATH}"
destination = "local/vars.env"
env = true
}
}
}
}
```