apache / apache/airflow

Better integration with systemd

Open
#16,846 6 comments 0 reactions 1 assignee Claimed by @yuqian90 View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

Airflow services (scheduler, worker and webserver) can be managed with systemd. Some small improvements will make this experience better:
- systemd has a feature called [WatchDogSec](https://www.freedesktop.org/software/systemd/man/systemd.service.html). It can be used to restart hung processes. To make use of the feature, the service needs to periodically call `sd_notify(3)`. It'll be nice to let the `heartbeat()` function call this so that systemd can monitor Airflow services.
- Airflow services by default writes a pid file to `AIRFLOW_HOME`. If the `AIRFLOW_HOME` is in a distributed file system, it causes some conflicts. E.g. when there are multiple airflow-worker or airflow-webserver being managed by systemd on different hosts. systemd already makes sure there's only one instance of the service running. So the pid file is redundant. It'll be nice to be able to turn off the pid file when systemd is used.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.