hashicorp / hashicorp/nomad

CSI: validate that single-node mounts aren't used with canaries

Open
#13,380 16 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/storage type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

1.3.1

### Operating system and Environment details

Debian 11.3

### Issue

Hi ✋

We're running Nomad 1.3.1 with 3 nomad masters, 3 nomad clients, consul, traefik and NFS plugin.

We are facing given allocation in pending state forever. It will end by progress_deadline (10m) and failed deployment.

I am not sure if it's related to CSI, we are using NFS (https://gitlab.com/rocketduck/csi-plugin-nfs). But maybe it's not, sometimes it's happing with CSI and sometimes without.

image

image

### Reproduction steps

Take a look at job file. If I change only metadata `version=10` to `version=20`, it will stuck. Pending until progress_deadline.

If I change port to static port or dynamic port, it does not matter. Sometimes it surprisely works. :-)

#### Expected Result

Deployment will success.

#### Actual Result

Deployment failed. Allocation is still pending.

### Job file (if appropriate)

```hcl
job "canary" {
type = "service"
datacenters = ["dc1"]

meta {
version = 10
}

update {
canary = 1
max_parallel = 1
health_check = "checks"
auto_revert = true
auto_promote = true
}

group "server" {
count = 1

network {
port "http" { to = 3001 }
}

volume "canary-data" {
type = "csi"
source = "canary-data-volume"
attachment_mode = "file-system"
access_mode = "single-node-writer"
}

task "echo" {
driver = "docker"
config {
image = "hashicorp/http-echo:latest"
args = [
"-listen", ":${NOMAD_PORT_http}",
"-text", "Hello world! IP ${NOMAD_IP_http} and PORT ${NOMAD_PORT_http}",
]
ports = ["http"]
}

resources {
cpu = 128
memory = 128
}

volume_mount {
volume = "canary-data"
destination = "/app/data"
}

service {
port = "http"

tags = [
"traefik.enable=true",
"traefik.http.routers.${NOMAD_JOB_ID}.rule=Host(`canary.domain.tld`)"
]

check {
type = "http"
path = "/"
interval = "10s"
timeout = "2s"
}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the behavior with the supplied Nomad job file, changing only the metadata version as described, and inspect the pending allocation and deployment progress deadline. Compare runs with the CSI volume and canary update settings; done means the issue has a documented validation or correction for single-node mounts used with canaries and the deployment reaches the expected successful state.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, docker
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.