HostManager network connection error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
This error occurred in a Host Manager running the lakeshore240 agents on the LAT at site during a network outage, causing the manager process to crash. The Host Manager stays online, and was recovered by restarting the manager process, at which point all managed agents were in the "down" state and had "down" for their target as well.
2023-08-30T20-06-19.033016 startup-op: launching manager
2023-08-30T20-06-19.033420 start called for manager
2023-08-30T20-06-19.034188 manager:0 Status is now "starting".
2023-08-30T20-06-19.035269 manager:0 Status is now "running".
2023-08-30T20-07-19.987573 manager:0 CRASH: [Failure instance: Traceback: <class 'RuntimeError'>: Could not run docker-compose or could not parse docker-compose file; exit code 1, error text: b'An HTTP request took too long to complete. Retry with --verbose to obtain debug information.\nIf you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).\n'
/usr/lib/python3/dist-packages/twisted/internet/defer.py:501:errback
/usr/lib/python3/dist-packages/twisted/internet/defer.py:568:_startRunCallbacks
/usr/lib/python3/dist-packages/twisted/internet/defer.py:654:_runCallbacks
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1475:gotResult
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/lib/python3/dist-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/ocs/.local/lib/python3.8/site-packages/ocs/agents/host_manager/agent.py:453:manager
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/lib/python3/dist-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/ocs/.local/lib/python3.8/site-packages/ocs/agents/host_manager/agent.py:134:_reload_config
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/lib/python3/dist-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/ocs/.local/lib/python3.8/site-packages/ocs/agents/host_manager/agent.py:100:_update_docker_services
/usr/lib/python3/dist-packages/twisted/internet/defer.py:1418:_inlineCallbacks
/home/ocs/.local/lib/python3.8/site-packages/ocs/agents/host_manager/drivers.py:369:parse_docker_state
]
2023-08-30T20-07-19.988809 manager:0 Status is now "done".
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ocs/agents/host_manager/agent.py at _reload_config, _update_docker_services, and manager, then read drivers.py at parse_docker_state. Reproduce or trace the docker-compose timeout path described in the log. Done should address the network-outage failure without leaving the Host Manager process crashed and verify the resulting agent state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100