Airflow 3.2.0: structured task logging increases log size by 1.89× and high-volume task runtime by 2.64×

### Under which category would you file this issue?

Task SDK

### Apache Airflow version

3.2.0

### What happened and how to reproduce it?

## What happened
I ran the same DAG and task workload on Apache Airflow 2.9.2 and 3.2.0.
The task emitted exactly the same payload:

- 100,001 payload lines
- 37.9 bytes of useful message content per line

Despite identical logical output, Airflow 3.2.0 produced substantially larger
task logs and took significantly longer to complete.

## Per-line record comparison

The following lines contain the same logical payload message.

Airflow 2.9.2:

[2026-08-16T13:54:02.571+0000 +0000] MainProcess MainThread {subprocess.py:93} INFO - Line No 100000: Airflow 2 Log Size Test

Airflow 3.2.0:

{"timestamp":"2026-08-16T17:43:20.929261Z","level":"info","event":"Line No 100000: Airflow 3 Log Size Test","logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook","filename":"subprocess.py","lineno":99}

The Airflow 3.2.0 record repeats a fully qualified logger identity for every line:

"logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"

The useful event message is unchanged, but the Airflow 3.2.0 representation adds structured fields for timestamp, level, logger, filename, and line number, as well as JSON syntax.

The Airflow 2.9.2 representation uses a compact source reference:

{subprocess.py:93}

The expanded logger identity is a significant source of repeated per-line metadata. It contributes to both storage amplification and the per-line
processing required to serialize and write task log records.

We are not claiming that the logger field alone explains the complete regression. It is one directly observable contributor to the broader structured-record overhead.

## Reproduction

The task emits 100,001 deterministic progress messages. The DAG, task code, payload, worker resources, executor, logging backend, and deployment settings were kept constant between the two tests. Only the Airflow version changed.

The task is representative of high-volume logging workloads.

## Results
──────────────────────┬───────────────┐
│ Metric │ AF 2.9.2 │ AF 3.2.0 │ Delta │
──────────────────────┼───────────────┤
│ Task log file size │ 12,493,072 B (11.91 MiB) │ 23,671,803 B (22.58 MiB) │ 1.89x │
──────────────────────┼───────────────┤
│ Payload lines emitted │ 100,001 │ 100,001 │ identical │
──────────────────────┼───────────────┤
│ Avg bytes per payload line │ 124.89 B │ 235.89 B │ +111.0 B/line │
──────────────────────┼───────────────┤
│ Useful message content per line │ 37.9 B │ 37.9 B │ identical │
──────────────────────┼───────────────┤
│ Signal ratio (content ÷ line) │ 30.3% │ 16.1% │ −14.2 pp │
──────────────────────┼───────────────┤
│ gzip -6 size │ 378,417 B │ 732,143 B │ 1.94x │
──────────────────────┼───────────────┤
│ Payload emit window │ 16.33 s │ 42.53 s* │ 2.60x │
──────────────────────┼───────────────┤
│ Sustained log throughput │ 6,123 lines/s │ 2,351 lines/s* │ −62% │
──────────────────────┼───────────────┤
│ Total task wall time │ 16.86 s │ 44.43 s* │ 2.64x │
──────────────────────┼───────────────┤
│ Framework (non-payload) lines │ 21 │ 259 │ 12.3x │
──────────────────────┼───────────────┤
│ Framework (non-payload) bytes │ 3,930 B │ 82,551 B │ 21.0x │
──────────────────────┴───────────────┘

## Impact

The Airflow 3.2.0 task log is approximately 89.5% larger in bytes and remains 1.94x larger after gzip compression.

The task takes approximately 163.5% longer overall, and sustained logging throughput falls by approximately 62%.

The impact affects more than remote storage:

- Increased local and remote log size.
- Increased upload and transfer volume.
- Higher CPU and serialization overhead.
- Slower access to completed task logs in the Airflow UI.
- Slower scrolling and log rendering for large logs.
- Potentially increased risk of incomplete or failed UI rendering.

### What you think should happen instead?

## Expected behavior

For identical task output, Airflow 3.x should not introduce a significant
runtime and storage regression solely because task logs are represented using
structured records.

At minimum, users should have a supported deployment-level option to choose a
compact persisted task-log representation without modifying operators or DAGs.

## Requested behavior

Please consider providing an opt-in compact or legacy task-log mode that:

1. Preserves the Airflow 3.x task-process/supervisor protocol.
2. Converts structured events to compact plaintext before persistence/upload,
if JSON is required internally.
3. Preserves timestamps, levels, tracebacks, multiline messages, ordering, and
task-attempt boundaries.
4. Works with worker pods and supported executors.
5. Remains compatible with the Airflow UI, API, local handlers, and remote
logging backends.
6. Does not require modifications to operators or provider code.

Possible configuration concepts:

[logging]
task_log_format = legacy

or:

[logging]
structured_task_logs = false

The exact setting name is not important; the required capability is an
explicit, documented way to control the persisted task-log representation.

## Related reports

- #50274: task log format appears to ignore configuration
- #53442: Airflow 3.x logging configuration behavior
- Discussion #53006: Task SDK logs sent as JSON to the supervisor
- #55173: large task logs become slower to scroll
- #58226: large logs not rendering in the UI
- #57896: large logs do not appear in Airflow 3.1.2
- #54717: Airflow log tail support
- #45516: unable to see logs in the web UI

### Operating System

Amazon Linux 2023, x86_64

### Deployment

Other

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

apache-airflow 3.2.0
apache-airflow-core 3.2.0
apache-airflow-providers-amazon 9.23.0
apache-airflow-providers-apache-hive 9.4.0
apache-airflow-providers-cncf-kubernetes 10.14.0
apache-airflow-providers-common-compat 1.14.1
apache-airflow-providers-common-io 1.7.1
apache-airflow-providers-common-sql 1.33.0
apache-airflow-providers-fab 3.5.0
apache-airflow-providers-git 0.2.4
apache-airflow-providers-http 6.0.0
apache-airflow-providers-postgres 6.6.1
apache-airflow-providers-smtp 2.4.3
apache-airflow-providers-standard 1.12.1
apache-airflow-task-sdk 1.2.0

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Start with the Task SDK structured events, the task-process/supervisor protocol, and the [logging] configuration path, then compare the related logging reports listed in the issue. Done means a supported, documented compact persisted-log option that preserves the listed metadata and remains compatible with the UI, API, local handlers, remote backends, workers, and executors.

Written by the indexing model from the issue text.