apache / apache/airflow

Windows support for Airflow

Open
#10,388 31 comments 9 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

**Description**

Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in https://pypi.org/project/python-daemon/), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (https://daemoniker.readthedocs.io/en/latest/) to launch tasks? What are the challenges and issues?

In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (https://github.com/microsoft/WSL/issues/5068).

**Use case / motivation**

Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.

**Possible implementation**

The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:

```python
# airflow/bin/cli.py
from daemoniker import Daemonizer

...

if args.daemon:
with Daemonizer() as (is_setup, daemonizer):
if is_setup:
pid, stdout, stderr, log_file = setup_locations("scheduler",
args.pid,
args.stdout,
args.stderr,
args.log_file)

_is_parent = daemonizer(
pid,
stdout_goto=stdout,
stderr_goto=stderr
)

job.run()
```

Contributor guide

Open the contributing guide

Research direction

Start with airflow/bin/cli.py and the daemon module usage for the scheduler and webserver, then review the proposed daemoniker example in the issue. Done would mean Airflow can run natively on Windows without WSL or Docker, with the scheduler and webserver daemon behavior addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.