Handling “Airflow confs” in Task SDK
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Figure out how to retrieve Airflow configurations (from `airflow.cfg`) in the Task SDK without importing Airflow core.
Investigate different approaches for configuration architecture in separated server and SDK components, analyze trade-offs, and recommend the best approach.
## Configuration Split Options to Investigate
### Option 1: Complete Independence
- **Approach**: Separate configuration files for server and SDK
- **Pros**: Clean separation, no network dependencies
- **Cons**: Configuration duplication, sync issues
- **Use case**: When server and workers are in different environments
### Option 2: API-Based Inheritance
- **Approach**: SDK queries server API for configuration values
- **Pros**: Centralized configuration management
- **Cons**: Network dependency, potential latency
- **Use case**: When workers need dynamic configuration updates
### Option 3: Bootstrap Injection
- **Approach**: Server injects essential config at task startup
- **Pros**: No runtime API calls, centralized control
- **Cons**: Static configuration, larger task payloads
- **Use case**: When configuration is stable and performance is critical
### Option 4: Hybrid Local-First + API Fallback
- **Approach**: Local config first, API fallback for inherited values, caching
- **Pros**: Performance + flexibility, graceful degradation
- **Cons**: Complexity, cache invalidation challenges
- **Use case**: Best of both worlds approach
## Open Questions
- What about `ProvidersManager` class that is used to fetch configs from Providers
Contributor guide
Assessment
This issue has not been assessed yet.