Update config parser to Load json config settings from files
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
Json config settings like `[core] dag_bundle_config_list` and `[secrets] backend_kwargs` can become quite large and cumbersome. The linux kernel sets a maximum total size limit of 256MB for environment variables, so adding large json blobs to the environment variables risks running into these limits. To work around this, json files could be loaded into the config using the same technique as secrets by specifying a path to a json file in the config and reading the file at runtime using `json.load()`.
### Use case/motivation
Like with config settings tagged as secrets, update the `conf.getjson()` method to also attempt to load json config settings from a path specified using a suffix, e.g. `dag_bundle_config_list_file=/opt/airflow/config/dag_bundles.json` would read the setting `dag_bundle_config_list` from `/opt/airflow/config/dag_bundles.json`
### Related issues
#65449
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at the conf.getjson() method and inspect the existing secret-file loading behavior, using related issue #65449 for context. Done means a *_file configuration value loads the corresponding JSON file at runtime while existing inline JSON settings continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100