apache / apache/airflow

Update config parser to Load json config settings from files

Open
#70,133 0 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Json config settings like `[core] dag_bundle_config_list` and `[secrets] backend_kwargs` can become quite large and cumbersome. The linux kernel sets a maximum total size limit of 256MB for environment variables, so adding large json blobs to the environment variables risks running into these limits. To work around this, json files could be loaded into the config using the same technique as secrets by specifying a path to a json file in the config and reading the file at runtime using `json.load()`.

### Use case/motivation

Like with config settings tagged as secrets, update the `conf.getjson()` method to also attempt to load json config settings from a path specified using a suffix, e.g. `dag_bundle_config_list_file=/opt/airflow/config/dag_bundles.json` would read the setting `dag_bundle_config_list` from `/opt/airflow/config/dag_bundles.json`

### Related issues

#65449

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at the conf.getjson() method and inspect the existing secret-file loading behavior, using related issue #65449 for context. Done means a *_file configuration value loads the corresponding JSON file at runtime while existing inline JSON settings continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.