docker / docker/cli

CLI hangs when creating a failing replicated-job with `--restart-condition=none`

Open
#2,979 5 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/swarm
Dominant language
Go
Stars
6.1k
Forks
2.2k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Description
When creating a replicated-job service, if the task fails with --restart-condition=none, when TotalCompletions has been reached the CLI hangs. As neither update_config or rollback_config can be set, it doesn't seem possible to control what Docker should do.

I'm using a replicated-job to run a migration during a deploy pipeline and ideally when all tasks have failed to converge, the docker service create|update would exit with an error.

Steps to reproduce the issue:

  1. docker service create --mode=replicated-job --replicas=1 --restart-condition=none bash "exit 1"

Describe the results you received:
The CLI command hangs indefinitely.

Describe the results you expected:
The CLI command to exit as it's reached its limit to attempt any more restarts of the tasks.

Output of docker version:

Docker version 20.10.3, build 48d30b5

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 74
  Running: 6
  Paused: 0
  Stopped: 68
 Images: 7
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: s6zwbsect4msdzs0fwv04nnq3
  Is Manager: true
  ClusterID: 570e06kxskw39x8d4fxetxpmc
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 143.198.18.91
  Manager Addresses:
   143.198.18.91:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-193-generic
 Operating System: Ubuntu 16.04.7 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 992.1MiB
 Name: adventure-text-01
 ID: 2KSA:3PPX:RJOD:YTJ6:AYLM:3NVB:QQXA:CIQN:ER3M:PTKX:OSFT:B7ED
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  provider=digitalocean
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the hang with the documented docker service create --mode=replicated-job --replicas=1 --restart-condition=none bash "exit 1" command, then trace the CLI's service create and task-convergence handling. Done means the command exits with an error after the job reaches its completion limit instead of waiting indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.