apache / apache/airflow

Add a way to skip the secret_backend

Open
#19,251 29 comments 7 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I use the gcp [secret_manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) as a secret_backend

2 problems :

- the implementation always first look for the secret_backend before trying the airflow variables , no way to skip the check to the secret_backend

something like
```python
Variable.get("totot",skip_secret_backend=True)
```

so change variable.py

```python
@classmethod
def get(
cls,
key: str,
default_var: Any = __NO_DEFAULT_SENTINEL,
deserialize_json: bool = False,
skip_secret_backend: bool = False,
) -> Any:
```

and also change the macro
```
{{ var.value.get('my.var', 'fallback') }}
```

- every variable that is not in the secret_backend but in the airflow variable will produce an ERROR log line , for some dag is really confusing to see at every run :

```
[2021-10-27 10:16:19,103] {secret_manager_client.py:93} ERROR - Google Cloud API Call Error (PermissionDenied): No access for Secret ID airflow-prod-variable-XXXXXX-XXXXX.
Did you add 'secretmanager.versions.access' permission?
```

Replace the log level ERROR to WARNING would be better since we don't know if the secret do not exist or if it's really a problem of access permission.

### Use case/motivation

Every Variable.get make a call to the secret_backend , would be great to make it configurable ( to first control the cost and the load on the secret_backend )

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with variable.py and the Variable.get call path, then inspect the macro usage and the secret_manager_client.py logging shown in the report. Confirm that callers can bypass the secret backend when requested and that missing or inaccessible secrets no longer produce the reported ERROR-level noise.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.