Add a way to skip the secret_backend
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
I use the gcp [secret_manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) as a secret_backend
2 problems :
- the implementation always first look for the secret_backend before trying the airflow variables , no way to skip the check to the secret_backend
something like
```python
Variable.get("totot",skip_secret_backend=True)
```
so change variable.py
```python
@classmethod
def get(
cls,
key: str,
default_var: Any = __NO_DEFAULT_SENTINEL,
deserialize_json: bool = False,
skip_secret_backend: bool = False,
) -> Any:
```
and also change the macro
```
{{ var.value.get('my.var', 'fallback') }}
```
- every variable that is not in the secret_backend but in the airflow variable will produce an ERROR log line , for some dag is really confusing to see at every run :
```
[2021-10-27 10:16:19,103] {secret_manager_client.py:93} ERROR - Google Cloud API Call Error (PermissionDenied): No access for Secret ID airflow-prod-variable-XXXXXX-XXXXX.
Did you add 'secretmanager.versions.access' permission?
```
Replace the log level ERROR to WARNING would be better since we don't know if the secret do not exist or if it's really a problem of access permission.
### Use case/motivation
Every Variable.get make a call to the secret_backend , would be great to make it configurable ( to first control the cost and the load on the secret_backend )
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with variable.py and the Variable.get call path, then inspect the macro usage and the secret_manager_client.py logging shown in the report. Confirm that callers can bypass the secret backend when requested and that missing or inaccessible secrets no longer produce the reported ERROR-level noise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100