apache / apache/airflow

Provide an interactive shell that allows using Airflow Connections

Open
#60,598 6 comments 1 reaction 0 assignees View on GitHub
area:CLI kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Many projects provide an interactive shell with all necessary bootstrapping done. For example Django's `django-admin shell` command.

It would be nice to have this on Airflow too. A command like `airflow shell` could be used.

It would make sense if the same kind of code that it used inside of tasks could be used interactively (e.g. not some parallel sdk API).

For example if you have a PythonOperator that calls a function like:
```
def pg_do_stuff(conn_id):
pg_hook = PostgresHook(postgres_conn_id=conn_id)
with pg_hook.get_sqlalchemy_engine().begin() as conn:
# do stuff
...
```

In our `airflow shell` command, you could similarly do:
```
> from airflow.providers.postgres.hooks.postgres import PostgresHook
> pg_hook = PostgresHook(postgres_conn_id="my_conn_id")
> conn = pg_hook.get_sqlalchemy_engine().connect()
> conn.execute(...)
```

### Use case/motivation

There are various use cases, here are two:

#### Production debugging
Given:
* you have access to a shell on your productive Airflow instance.
* you have some connections (DB, HTTP, whatever) that are used in existing Airflow DAGs, that, due to firewall restrictions, you can not access from your workstation.
* a previously working DAG stops delivering the data you expect it to. Perhaps there's an error, or perhaps the data received simply doesn't look like it is supposed to.

Then:
It would be nice to be able to connect to a python shell on the prod (or integration) Airflow instance and interactively run commands, use the defined connections, and investigate what is happening.

#### Local development
Given:
* you are developing a task to pull and process data from a data source
* you are not familiar with the data delivered

Then:
It would be nice to have an interactive python shell that allows you to use that connection interactively.

### Related issues

Discussion: https://github.com/apache/airflow/discussions/60334

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading the related discussion at https://github.com/apache/airflow/discussions/60334 and reviewing how Airflow CLI commands are organized. Define the shell command's bootstrap behavior and how existing provider hooks and configured Connections are exposed interactively. Done means the proposed command supports the described connection-based debugging and development workflows with documented tests and usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.