globus / globus/globus-compute

funcx-endpoint can start a duplicate endpoint on multi-login-node systems

Open
#581 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

**Describe the bug**
`funcx-endpoint list` relies on checking for the PID listed in the `daemon.pid` file to determine whether the endpoint is live. On multi-login-node systems this check will fail because the process might be on a different login node. The issue here is that the `funcx-endpoint list` will erroneously say the endpoint is disconnected, and then when the user tries to start the endpoint, the cli will wipe the current daemon.pid file in an attempt to cleanup, and then start a new endpoint with the same endpoint_id, ending up in a broken state.

**To Reproduce**
Steps to reproduce the behavior, for e.g:
1. Install funcx-endpoint==0.3.2 with Python 3.7/3.8 on cluster
2. Connect to loginnode01 of many
3. Run `funcx-endpoint configure test; funcx-endpoint start test`
4. Connect to loginnode02
5. Run `funcx-endpoint list`; This will show `test` is `disconnected`
6. Run `funcx-endpoint start test`.

**Expected behavior**
`funcx-endpoint list` should not show a connected endpoint on another login node as `disconnected`.
`funcx-endpoint start` should not wipe the daemon.pid, and start a duplicate endpoint with the same endpoint id.

**Distributed Environment**
- Running on a multi-login-node system

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `funcx-endpoint list` and `funcx-endpoint start` commands, focusing on how they use and clean up `daemon.pid` on multi-login-node systems. Reproduce the sequence across two login nodes; done means `list` does not report the remote endpoint as disconnected and `start` does not remove the existing PID file or launch a duplicate endpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.