microsoft / microsoft/hcsshim

Failed to shutdown container with hcsshim error "WaitTimeout: hcsshim: timeout waiting for notification"

Open
#1,056 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
694
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
28

Description

Hi, I am running Kubernete on Windows, and runtime is docker. When I delete Windows Pod, following errors are printed

time="2021-06-29T18:50:48.483759300-07:00" level=error msg="hcsshim::ComputeSystem::WaitTimeout - End Operation - Error" cid=d775ab1799c663245df
b553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb error="WaitTimeout d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb: hcsshim: time
out waiting for notification"                                                                                                                   
time="2021-06-29T18:50:48.483759300-07:00" level=error msg="failed to shutdown container, and subsequent terminate also failed" container=d775ab
1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb error="container d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb enc
ountered an error during WaitTimeout: hcsshim: timeout waiting for notification" module=libcontainerd namespace=moby                            
time="2021-06-29T18:50:48.483759300-07:00" level=warning msg="failed to shutdown container" container=d775ab1799c663245dfb553d6667bd27f7bb3c39ff
7786cc55f21dc91555cadb error="container d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb encountered an error during WaitTimeout
: hcsshim: timeout waiting for notification: subsequent terminate failed container d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555c
adb encountered an error during WaitTimeout: hcsshim: timeout waiting for notification" module=libcontainerd namespace=moby process=init        
time="2021-06-29T18:50:48.483759300-07:00" level=info msg="sending event" container=d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555
cadb event=exit event-info="{d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb init 816 4294967295 2021-06-29 18:40:48.4397434 -0
700 PDT m=+1797.570635201 false failed to shutdown container: container d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb encount
ered an error during WaitTimeout: hcsshim: timeout waiting for notification: subsequent terminate failed container d775ab1799c663245dfb553d6667b
d27f7bb3c39ff7786cc55f21dc91555cadb encountered an error during WaitTimeout: hcsshim: timeout waiting for notification}" module=libcontainerd na
mespace=moby                                                                                                                                    
time="2021-06-29T18:50:48.483759300-07:00" level=error msg="failed to terminate hccshim container" container=d775ab1799c663245dfb553d6667bd27f7b
b3c39ff7786cc55f21dc91555cadb error="container d775ab1799c663245dfb553d6667bd27f7bb3c39ff7786cc55f21dc91555cadb encountered an error during Term
inate: hcsshim: the handle has already been closed" module=libcontainerd namespace=moby pid=816 process=init signal=9

The error might cause kubelet fail to sync message to kube-apiserver, and cause the Windows Node status to be not ready.

My Windows version is Windows Server 2019 (1809), and docker version is 19.03.5

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported hcsshim WaitTimeout and subsequent handle-closed errors while deleting a Windows pod, using the stated Windows Server 2019 (1809) and Docker 19.03.5 versions. Reproduce the container shutdown failure and determine whether it explains the kubelet synchronization and Windows Node readiness symptoms; done requires a confirmed cause and a validated fix or workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.