apache / apache/airflow

FAB Providers API get_application_builder cache miss trigger init_appbuilder

Open
#72,937 1 comment 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:fab
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.0.1

### What happened and how to reproduce it?

I have a script to sync some of our internal roles into FAB API, this will calls `roles` and `users` api multiple times to sync into Airflow FAB

During one of such script run, I noticed that calling FAB roles or users API multiple times (in around 1000 times over 2 hours) will eventually lead to create memory issue, k8s pod seems to increase memory usage into more than 50 GBs and this leads to degradation of API server and causing either OOM killed or slow response time (up to 10s) eventually lead to liveness probe failure

Each API call seems to run providers/fab/src/airflow/providers/fab/auth_manager/cli_commands/utils.py get_application_builder function this will:
- Create a new Flask app
- yield cached return_app_builder

When cache is missed, it will call init_appbuilder and I think Python store the cache in memory
Since the cache is always missing it will eventually causing memory issue

### What you think should happen instead?

Calling API should not call init_appbuilder every single time, so either API calls need to store the created flask app object in memory or just remove cache entirely if it's expected to init every time

### Operating System

Debian GNU/Linux 12 (bookworm)

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

fab

### Versions of Apache Airflow Providers

3.7.0

### Official Helm Chart version

1.18.0

### Kubernetes Version

1.33.5

### Helm Chart configuration

Not Applicable

### Docker Image customizations

Not Applicable

### Anything else?

Occur every time

One way to replicate
- Using helm chart with admin user and default user admin enabled
- Go inside the pod and authenticate with the admin user (using username and password) through auth/token endpoint
- Use the token generated from the login method
- Call get `auth/fab/v1/roles`
- Check logs to see that it triggers initialize_app_builder with logs `Initializing AppBuilder` and `Registering Route`
- Repeat API calls

Example

```
curl -v -X POST "http://127.0.0.1:8080/auth/token" -H "Content-Type: application/json" -d '{"username": "admin", "password": "admin"}'
{"access_token":"asdf"}

TOKEN='asdf'

curl -X GET "http://localhost:8080/auth/fab/v1/roles" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json"

```

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in providers/fab/src/airflow/providers/fab/auth_manager/cli_commands/utils.py at get_application_builder and follow its calls to init_appbuilder. Reproduce repeated authenticated GET requests to auth/fab/v1/roles or users, watching the “Initializing AppBuilder” and “Registering Route” logs and pod memory. Done means repeated API calls no longer repeatedly initialize builders or cause unbounded memory growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
flask, helm, kubernetes, python
Domain
api, authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.