DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Hi everyone,
### Description
I recently came across the problem, that the git-fetch-requests of the dag-bundle hit the rate-limit of the git-server.
I found out that Airflow cannot **take advantage of a higher rate-limit for authorized git-requests** if the repo is public as it never sends the credentials to the server.
But Airflow could support to **force authentication** for a git connection type via a custom http.extraHeader as additional argument to the git command. This would allow authenticated Git operations even for public repositories.
### Use case/motivation
Git does not authenticate when syncing public repositories, even if login and password are set. This is due to Git’s HTTPS behavior: **it always tries anonymous access first** and only sends credentials if the server responds with a 401 Unauthorized.
=> Public repos typically do not issue a 401, so credentials are never used.
This is problematic because GitLab/GitHub apply different rate limits for authenticated vs anonymous requests. As a result, Airflow tasks can hit anonymous rate limits even when valid credentials are provided.
**A known workaround** is to use Git’s http.extraHeader option to force authentication:
```
B64=$(printf 'x:%s' "" | base64)
git -c http.extraHeader="Authorization: Basic $B64" clone https://gitlab.com//.git
```
This ensures that all requests are authenticated from the first request, avoiding anonymous rate limits.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
- [x] I am still thinking about a possible solution ...
This functionality probably could be added here 🤔
https://github.com/apache/airflow/blob/main/providers/git/src/airflow/providers/git/hooks/git.py
Perhaps by additional extra arguments
- `config-env` : which represents the git argument `--config-env` - https://git-scm.com/docs/git/2.34.0#Documentation/git.txt---config-envnameenvvar
- `env` : which allows to extent the environment in which git gets executed
This would allow to pass the authentication header to git like you would do it in the shell:
```
export MY_HEADER="Authorization: Basic "
git --config-env=http.extraheader=MY_HEADER clone
```
This would be a very general approach which could also be used for other git-features. But of course it is less intuitiv.
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in providers/git/src/airflow/providers/git/hooks/git.py and review how Git commands receive connection credentials and extra arguments. Reproduce syncing a public repository with credentials, then add coverage showing that forced authentication avoids anonymous rate limits while preserving existing Git connection behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100