Documentation: How to correctly wrap scheduler and workers into systemd unit files as services?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have been investigating how to wrap both the scheduler process as well as the worker processes into systemd services via unit files. The procedure is not really documented. My understanding is that taking care or generating unit files is not Dask's job - so far so good. But the "correct" configuration of a service and Dask's requirements in this regard should be documented.
I found two examples, [here](https://github.com/nlesc-sherlock/emma/blob/master/roles/dask/templates/dask-scheduler.systemd.j2) and [here](https://github.com/nlesc-sherlock/emma/blob/master/roles/dask/templates/dask-worker.systemd.j2). I have been trying to build on top of that and to generalize this to a point where I can run Dask out of a miniforge environment.
What I came up with, working so far, looks as follows (templates):
```ini
[Unit]
Description=Dask Scheduler
After=syslog.target network.target
[Service]
Type=simple
WorkingDirectory=$HOME
User=${PREFIX}user
Environment="PATH=${FORGE}/envs/${PREFIX}env/bin:${PATH}"
ExecStart=${FORGE}/envs/${PREFIX}env/bin/python ${FORGE}/envs/${PREFIX}env/bin/dask-scheduler \
--pid-file=$HOME/scheduler.pid \
--protocol tls \
--tls-ca-file $HOME/${PREFIX}_ca.crt \
--tls-cert $HOME/${PREFIX}_node.crt --tls-key $HOME/${PREFIX}_node.key \
--port $PORT --dashboard-address $DASHPORT
ExecStop=/bin/kill `/bin/cat $HOME/scheduler.pid`
[Install]
WantedBy=default.target
```
and
```ini
[Unit]
Description=Dask Worker
After=syslog.target network.target
[Service]
Type=simple
WorkingDirectory=$HOME
User=${PREFIX}user
Environment="PATH=${FORGE}/envs/${PREFIX}env/bin:${PATH}"
ExecStart=${FORGE}/envs/${PREFIX}env/bin/python ${FORGE}/envs/${PREFIX}env/bin/dask-worker \
--pid-file=$HOME/worker.pid \
--protocol tls \
--tls-ca-file $HOME/${PREFIX}_ca.crt \
--tls-cert $HOME/${PREFIX}_node.crt --tls-key $HOME/${PREFIX}_node.key \
--dashboard-address $DASHPORT --nanny-port $NANNY \
--worker-port $PORT \
tls://$SCHEDULER:$PORT
ExecStop=/bin/kill `/bin/cat $HOME/worker.pid`
[Install]
WantedBy=default.target
```
It raises two questions:
- Does the above make any sense?
- Is the type "simple" the right type? Systemd also supports "forking" for services that do generate forks. It is my understanding that Dask can do this depending on its mode of operation (i.e. what the user does with it). If I set this value to "forking", the services fail to start with timeout exceptions. I am guessing that systemd's assumption is that the process start also involves a distinct fork (i.e. the process does not only creates forks as part of its longer-term operation but also initializes itself via a fork or daemon-like double-fork).
This issue is related to #496 and #497.
Contributor guide
Assessment
This issue has not been assessed yet.