simonsobs / simonsobs/ocs

HostManager failed to remove container

Open
#373 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

agent: hostmanager bug needs triage
Dominant language
Python
Stars
19
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Odd problem today (finally resolved at 2024-01-18 14:30:06 UTC) when ACU agent was misbehaving, and hostmanager "down" could not kill it (twice, though first attempt was brief). HM logs show:

2024-01-18T14:23:51+0000 start called for update
2024-01-18T14:23:51+0000 update:140 Status is now "starting".
2024-01-18T14:23:51+0000 update:140 Status is now "running".
2024-01-18T14:23:51+0000 update:140 Update requested.
2024-01-18T14:23:51+0000 update:140 Status is now "done".
2024-01-18T14:23:52+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:23:57+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:23:58+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:03+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:08+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:24:09+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:13+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:19+0000 manager:0 Agent instance ACUAgent:acu refused to die.
...

But running docker-compose rm --stop --force ocs-acu, as ocs user, did bring down the container, without any issues. (That command should be exactly what the HM runs.)

I have no useful analysis yet...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the HostManager shutdown path that runs docker-compose rm --stop --force ocs-acu, then compare it with the logged termination attempts and the successful manual command. Reproduce the ACU agent failure if possible; done means HostManager reliably removes the container when the agent refuses to terminate.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.