apache / apache/airflow

Provider migration for decoupling remote logging from core

Open
#70,265 10 comments 0 reactions 0 assignees View on GitHub
area:logging area:providers kind:meta on hold
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

- related: #67056

## Why

#67056 decoupled remote logging from the hardcoded if/elif chain in
`airflow-core/src/airflow/config_templates/airflow_local_settings.py`. Core and the Task SDK now
share one resolver (`airflow_shared.logging.factory.resolve_remote_task_log`) that dispatches on the
URL scheme of `[logging] remote_base_log_folder` via `ProvidersManager` and instantiates the
provider class through a no-arg `from_config()` classmethod.

The legacy per-scheme chain in `airflow_local_settings.py` is transitional: it stays as a fallback
until every remote logging backend has been migrated to the new mechanism. This issue tracks that
migration, one sub-issue per backend.

## The pattern

Each migration is a small, self-contained PR with three parts (see the merged amazon examples:
s3 — #69817, cloudwatch — #69816):

1. **Add `from_config()`** — a no-arg classmethod on the provider's `RemoteLogIO` class that mirrors
the corresponding legacy branch in `airflow_local_settings.py`, including the
`[logging] remote_task_handler_kwargs` IO-kwargs merge (where the legacy branch applies it) and
`expanduser` on `base_log_folder`, so behavior is unchanged for existing configs.
2. **Register the scheme** — add a `remote-logging:` section to the provider's `provider.yaml` and
mirror it in `get_provider_info.py`:

```yaml
remote-logging:
- classpath: airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO
scheme: s3
```

3. **Tests** — mirror `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`: `from_config` field mapping,
IO-kwargs filtering, scheme registration, and end-to-end `resolve_remote_task_log` dispatch.
4. **Real system test** — set up the actual backend yourself (a real service or a local
equivalent), run a task with remote logging enabled, and verify logs are uploaded and read back
through the new dispatch path. Include the setup and verification results in the PR description.

If `from_config()` raises on a bad config, the shared factory falls back to the legacy path — so
these migrations are non-breaking by construction.

## Backends

- [x] `s3` (amazon) — #69817
- [x] `cloudwatch` (amazon) — #69816
- [x] `gs` (google) — #70266
- [x] `stackdriver` (google) — #70267
- [x] `wasb` (microsoft.azure) — #70268
- [x] `oss` (alibaba) — #70269
- [x] `hdfs` (apache.hdfs) — #70270
- [x] `elasticsearch` — #70271
- [x] `opensearch` — #70272

## Follow-up once backends are migrated

- [ ] Once all provider migrations are merged **and released**, add a deprecation warning to the
legacy branches in `airflow_local_settings.py`, gated on the provider versions that ship
`from_config` (step 2 of the plan in #67056).
- [ ] Remove the hardcoded remote logging config from core in a future minor or major release,
whichever the community prefers.

## How to contribute

Comment on a sub-issue to claim it — one PR per scheme, following the merged amazon examples.
The `elasticsearch` and `opensearch` sub-issues include a small design decision (those backends are
currently selected by host config rather than URL scheme); design feedback is welcome there.

Contributor guide

Open the contributing guide

Research direction

Read airflow-core/src/airflow/config_templates/airflow_local_settings.py and airflow_shared.logging.factory.resolve_remote_task_log, then compare the merged provider examples and their provider.yaml and get_provider_info.py entries. The listed backend migrations are already checked off; remaining work is to coordinate the post-release deprecation warning and eventual removal of the legacy core configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.