globus / globus/globus-compute

Workers left alive on MOM node

Open
#780 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

**Describe the bug**
Workers are not cleaned up from a launch node after a job is terminated by the schedular.

My app places the FuncX manager on the MOM node of a Cray supercomputer so that workers can launch MPI applications via system calls. The manager process is killed when the job exits but the workers stay afterwards.

Looking at the logs, I note the workers report receiving a Signal 15 but do not exit. Is that expected?

```
(miniconda-3/latest//home/lward/exalearn/edw/env) lward@thetalogin6:~/.funcx/nwchem/HighThroughputExecutor/worker_logs/70f647195873> more funcx_worker_32.log
1649883351.811096 2022-04-13 20:55:51 INFO MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:85 __init__ Initializing worker 32
1649883351.813936 2022-04-13 20:55:51 INFO MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:86 __init__ Worker is of type: RAW
1649883351.814639 2022-04-13 20:55:51 INFO MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:95 __init__ Trying to connect to : tcp://127.0.0.1:52075
1649883351.815704 2022-04-13 20:55:51 INFO MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:109 start Starting worker
1649884526.686283 2022-04-13 21:15:26 ERROR MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:101 handler Signal handler called with signal 15
1649884762.299373 2022-04-13 21:19:22 ERROR MainProcess-70029 MainThread-140634291058496 funcx_endpoint.executors.high_throughput.funcx_worker:101 handler Signal handler called with signal 15
```

**To Reproduce**
TBD. My app has a complex set up, but I can create a minimal example on request.

**Expected behavior**
Everything dies when Cobalt commands it.

**Environment**
- OS: CentOS
- OS & Container technology: None
- Python version @ 3.8.12

- Python version @ 3.8.12
- funcx version @ 58493f
- funcx-endpoint version @ 58493f

**Distributed Environment**
- Where are you running the funcX script from? Login node
- Where does the endpoint run? Login node
- What is your endpoint-uuid? ff59d7d1-e2f5-4a38-8bb8-ba6de588c7c7
- Attach endpoint logs at `~/.funcx/` if this is an endpoint issue.
Please let us know if you'd prefer to share logs privately.

[worker-no-die.tar.gz](https://github.com/funcx-faas/funcX/files/8484810/worker-no-die.tar.gz)

Contributor guide

Open the contributing guide

Research direction

Start with funcx_endpoint.executors.high_throughput.funcx_worker and the worker log showing repeated Signal 15 handling. Reproduce the worker cleanup behavior in a minimal setup, then verify that workers exit when the manager job is terminated by Cobalt.

Written by the indexing model from the issue text.

Assessment

Tech stack
centos, python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.