globus / globus/globus-compute
funcx-endpoint can start a duplicate endpoint on multi-login-node systems
- Dominant language
- Python
- Stars
- 162
- Forks
- 53
- Avg merge
- 15h 29m
- Merged PRs (30d)
- 26
Description
**Describe the bug**
`funcx-endpoint list` relies on checking for the PID listed in the `daemon.pid` file to determine whether the endpoint is live. On multi-login-node systems this check will fail because the process might be on a different login node. The issue here is that the `funcx-endpoint list` will erroneously say the endpoint is disconnected, and then when the user tries to start the endpoint, the cli will wipe the current daemon.pid file in an attempt to cleanup, and then start a new endpoint with the same endpoint_id, ending up in a broken state.
**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Install funcx-endpoint==0.3.2 with Python 3.7/3.8 on cluster
2. Connect to loginnode01 of many
3. Run `funcx-endpoint configure test; funcx-endpoint start test`
4. Connect to loginnode02
5. Run `funcx-endpoint list`; This will show `test` is `disconnected`
6. Run `funcx-endpoint start test`.
**Expected behavior**
`funcx-endpoint list` should not show a connected endpoint on another login node as `disconnected`.
`funcx-endpoint start` should not wipe the daemon.pid, and start a duplicate endpoint with the same endpoint id.
**Distributed Environment**
- Running on a multi-login-node system
Contributor guide
Research direction
Start by tracing the `funcx-endpoint list` and `funcx-endpoint start` commands, focusing on how they use and clean up `daemon.pid` on multi-login-node systems. Reproduce the sequence across two login nodes; done means `list` does not report the remote endpoint as disconnected and `start` does not remove the existing PID file or launch a duplicate endpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100