apache / apache/airflow

Handling “Airflow confs” in Task SDK

Open
#44,352 7 comments 0 reactions 1 assignee Assigned to @amoghrajesh View on GitHub
area:core area:task-execution-interface-aip72 priority:medium
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

Figure out how to retrieve Airflow configurations (from `airflow.cfg`) in the Task SDK without importing Airflow core.

Investigate different approaches for configuration architecture in separated server and SDK components, analyze trade-offs, and recommend the best approach.

## Configuration Split Options to Investigate

### Option 1: Complete Independence
- **Approach**: Separate configuration files for server and SDK
- **Pros**: Clean separation, no network dependencies
- **Cons**: Configuration duplication, sync issues
- **Use case**: When server and workers are in different environments

### Option 2: API-Based Inheritance
- **Approach**: SDK queries server API for configuration values
- **Pros**: Centralized configuration management
- **Cons**: Network dependency, potential latency
- **Use case**: When workers need dynamic configuration updates

### Option 3: Bootstrap Injection
- **Approach**: Server injects essential config at task startup
- **Pros**: No runtime API calls, centralized control
- **Cons**: Static configuration, larger task payloads
- **Use case**: When configuration is stable and performance is critical

### Option 4: Hybrid Local-First + API Fallback
- **Approach**: Local config first, API fallback for inherited values, caching
- **Pros**: Performance + flexibility, graceful degradation
- **Cons**: Complexity, cache invalidation challenges
- **Use case**: Best of both worlds approach

## Open Questions

- What about `ProvidersManager` class that is used to fetch configs from Providers

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.