apache / apache/airflow

PostgreSQL Database Backend Section Requires some additional info on fallback hosts

Open
#25,461 4 comments 0 reactions 0 assignees View on GitHub
kind:documentation
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### What do you see as an issue?

1. Airflow uses SQLAlchemy - which in turn uses Psycopg2 for database communication with PostgreSQL backend
2. It is possible to use multiple database hosts where a High Availability (HA) is required - i.e. a primary database host and one/multiple secondary hosts for a postgreSQL ecosystem. The objective is - when and if the primary database host is unavailable, the connection is retried with the newly-elected primary host to restore database connectivity.
3. Airflow documentation - https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html - should provide some indication to where to consult for further info, so that users of Airflow has a better experience in troubleshooting issues with database setup.

### Solving the problem

1. The proposal is to add an additional section under this - https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html
2. The section will be marked with "Note"
3. The section will have the following text

> It is possible to have multiple fallback hosts in PostgreSQL so that any loss of primary hosts does not impact your Airflow operation. Airflow currently uses SQLAlchemy for database session management. Please refer to their documentation [here](https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#specifying-multiple-fallback-hosts) for more information.
known issue in SQLAlchemy - https://github.com/sqlalchemy/sqlalchemy/issues/4392 (Related to multiple host URL specification)

### Anything else

1. The issue currently occurs for any Airflow version, especially with multiple fallback hosts for metadata DBs
2. There is an ongoing [bugfix ](https://github.com/sqlalchemy/sqlalchemy/issues/4392)in SQLAlchemy which will ensure that:
a. URL Specification in psycopg2 is implemented correctly
b. Documentation in SQLAlchemy is consistent, so that users do not get mislead
3. I am requesting a thorough code review at airflow side, as I am not a Python developer, and do not fully understand how the code is written. I am happy to submit a PR on documentation change, if this ticket is considered valuable.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with Airflow's set-up-database documentation page and review the PostgreSQL backend section. Add a Note directing users to SQLAlchemy's multiple-fallback-host documentation and retain the SQLAlchemy issue link for context; done means the section clearly explains the fallback-host guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sqlalchemy
Domain
databases, documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.