apache / apache/airflow

DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues

Open
#54,829 24 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue kind:bug provider:git
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

Hi everyone,

### Description

I recently came across the problem, that the git-fetch-requests of the dag-bundle hit the rate-limit of the git-server.
I found out that Airflow cannot **take advantage of a higher rate-limit for authorized git-requests** if the repo is public as it never sends the credentials to the server.

But Airflow could support to **force authentication** for a git connection type via a custom http.extraHeader as additional argument to the git command. This would allow authenticated Git operations even for public repositories.

### Use case/motivation

Git does not authenticate when syncing public repositories, even if login and password are set. This is due to Git’s HTTPS behavior: **it always tries anonymous access first** and only sends credentials if the server responds with a 401 Unauthorized.
=> Public repos typically do not issue a 401, so credentials are never used.

This is problematic because GitLab/GitHub apply different rate limits for authenticated vs anonymous requests. As a result, Airflow tasks can hit anonymous rate limits even when valid credentials are provided.

**A known workaround** is to use Git’s http.extraHeader option to force authentication:
```
B64=$(printf 'x:%s' "" | base64)
git -c http.extraHeader="Authorization: Basic $B64" clone https://gitlab.com//.git
```
This ensures that all requests are authenticated from the first request, avoiding anonymous rate limits.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!
- [x] I am still thinking about a possible solution ...

This functionality probably could be added here 🤔
https://github.com/apache/airflow/blob/main/providers/git/src/airflow/providers/git/hooks/git.py
Perhaps by additional extra arguments
- `config-env` : which represents the git argument `--config-env` - https://git-scm.com/docs/git/2.34.0#Documentation/git.txt---config-envnameenvvar
- `env` : which allows to extent the environment in which git gets executed

This would allow to pass the authentication header to git like you would do it in the shell:
```
export MY_HEADER="Authorization: Basic "
git --config-env=http.extraheader=MY_HEADER clone
```
This would be a very general approach which could also be used for other git-features. But of course it is less intuitiv.

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in providers/git/src/airflow/providers/git/hooks/git.py and review how Git commands receive connection credentials and extra arguments. Reproduce syncing a public repository with credentials, then add coverage showing that forced authentication avoids anonymous rate limits while preserving existing Git connection behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.