A service update using the stop-first update policy does not respect the existing task’s stop grace period when it exceeds one minute.
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Description
When a service definition specifies a stop_grace_period longer than one minute and uses a stop-first update order, the stop-first semantics are violated if the existing task takes more than one minute to stop. The new task is started after no more than a one-minute delay.
Expected behavior
stop_grace_period is respected and "stop-first" semantics is respected.
Steps to reproduce
stack.yml
version: '3.6'
services:
waiting-service:
image: alpine
stop_grace_period: 90s
deploy:
mode: replicated
replicas: 1
update_config:
order: stop-first
entrypoint: ash
command:
- -c
- |
trap 'echo "received TERM"' TERM
echo "Up"
while true; do echo "Still up"; sleep 10; done
docker stack deploy -c stack.yml testconv
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
Creating network testconv_default
Creating service testconv_waiting-service
date; docker service update --force --detach=false testconv_waiting-service; date
mer 29 lug 2026 19:52:26 CEST
testconv_waiting-service
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service testconv_waiting-service converged
mer 29 lug 2026 19:53:32 CEST
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
iixvooki30e1 testconv_waiting-service replicated 2/1 alpine:latest
docker service logs -t testconv_waiting-service
2026-07-29T17:51:54.290743926Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Up
2026-07-29T17:51:54.290766426Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:04.293327125Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:14.296006296Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:24.297436301Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:34.301911917Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | received TERM
2026-07-29T17:52:34.301936792Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:44.302612255Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:52:54.303608509Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:04.304309375Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:14.304741588Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:24.306502551Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:27.190412177Z testconv_waiting-service.1.wjnandmyd9nt@docker-desktop | Up
2026-07-29T17:53:27.190482552Z testconv_waiting-service.1.wjnandmyd9nt@docker-desktop | Still up
2026-07-29T17:53:34.308361792Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:37.194422251Z testconv_waiting-service.1.wjnandmyd9nt@docker-desktop | Still up
2026-07-29T17:53:44.309728963Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:47.195115131Z testconv_waiting-service.1.wjnandmyd9nt@docker-desktop | Still up
2026-07-29T17:53:54.311676926Z testconv_waiting-service.1.7zsxoyo6rp6p@docker-desktop | Still up
2026-07-29T17:53:57.199606844Z testconv_waiting-service.1.wjnandmyd9nt@docker-desktop | Still up
Considerations
Looking at the source code (I'm not familiar with it or with Go, so apologies if my hypothesis is incorrect), the root cause seems to be a mismatch between stop_grace_period and the TaskTimeout value used by the updater orchestrator.
In manager/orchestrator/restart/restart.go, TaskTimeout is unconditionally hardcoded to one minute:
const defaultOldTaskTimeout = time.Minute
func NewSupervisor(...) *Supervisor {
return &Supervisor{
TaskTimeout: defaultOldTaskTimeout,
}
}
In stop-first mode, the updater calls DelayStart with the old task and waitStop=true. Inside DelayStart, a timer is created using r.TaskTimeout, rather than stop_grace_period. As a result, the manager waits at most one minute for the old task to stop, then proceeds to start the new one regardless.
stop_grace_period is never read by the restart supervisor. It is only passed to the agent side and therefore has no effect on how long the Swarm manager waits at the orchestration level.
The practical impact is that, when stop_grace_period is greater than one minute, the manager times out and creates the new container while the old one is still running. As a result, the old and new containers coexist, violating the stop-first semantics.
I also posted a similar comment on the existing issue https://github.com/moby/moby/issues/41380. However, since the original issue has received little attention, I’m trying to determine whether the SwarmKit repository would be a more appropriate place to report it.
Engine version
Engine:
Version: 29.6.2
API version: 1.55 (minimum version 1.40)
Go version: go1.26.5
Git commit: 3d80467
Built: Thu Jul 16 16:13:03 2026
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v2.2.5
GitCommit: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
runc:
Version: 1.3.6
GitCommit: v1.3.6-0-g491b69ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in manager/orchestrator/restart/restart.go, focusing on NewSupervisor, TaskTimeout, and DelayStart, then reproduce the behavior with the provided stack.yml and stop-first update command. Trace how stop_grace_period reaches orchestration and verify that the old task remains alone until its grace period ends; done means no overlapping old and new tasks during the update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100