FAB Providers API get_application_builder cache miss trigger init_appbuilder
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.0.1
### What happened and how to reproduce it?
I have a script to sync some of our internal roles into FAB API, this will calls `roles` and `users` api multiple times to sync into Airflow FAB
During one of such script run, I noticed that calling FAB roles or users API multiple times (in around 1000 times over 2 hours) will eventually lead to create memory issue, k8s pod seems to increase memory usage into more than 50 GBs and this leads to degradation of API server and causing either OOM killed or slow response time (up to 10s) eventually lead to liveness probe failure
Each API call seems to run providers/fab/src/airflow/providers/fab/auth_manager/cli_commands/utils.py get_application_builder function this will:
- Create a new Flask app
- yield cached return_app_builder
When cache is missed, it will call init_appbuilder and I think Python store the cache in memory
Since the cache is always missing it will eventually causing memory issue
### What you think should happen instead?
Calling API should not call init_appbuilder every single time, so either API calls need to store the created flask app object in memory or just remove cache entirely if it's expected to init every time
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
fab
### Versions of Apache Airflow Providers
3.7.0
### Official Helm Chart version
1.18.0
### Kubernetes Version
1.33.5
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Not Applicable
### Anything else?
Occur every time
One way to replicate
- Using helm chart with admin user and default user admin enabled
- Go inside the pod and authenticate with the admin user (using username and password) through auth/token endpoint
- Use the token generated from the login method
- Call get `auth/fab/v1/roles`
- Check logs to see that it triggers initialize_app_builder with logs `Initializing AppBuilder` and `Registering Route`
- Repeat API calls
Example
```
curl -v -X POST "http://127.0.0.1:8080/auth/token" -H "Content-Type: application/json" -d '{"username": "admin", "password": "admin"}'
{"access_token":"asdf"}
TOKEN='asdf'
curl -X GET "http://localhost:8080/auth/fab/v1/roles" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json"
```
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in providers/fab/src/airflow/providers/fab/auth_manager/cli_commands/utils.py at get_application_builder and follow its calls to init_appbuilder. Reproduce repeated authenticated GET requests to auth/fab/v1/roles or users, watching the “Initializing AppBuilder” and “Registering Route” logs and pod memory. Done means repeated API calls no longer repeatedly initialize builders or cause unbounded memory growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- flask, helm, kubernetes, python
- Domain
- api, authentication, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100