Logs not uploaded to S3 on failed retries

### Apache Airflow version

3.1.0

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

We are experiencing a critical issue with log upload to S3 when using DbtTaskGroup from Cosmos in our Airflow 3.x environment (Cosmos 1.10.2, dbt-core 1.11.x, Astronomer runtime). The airflow deployment is running on EKS in AWS and tasks in airflow are run as KubernetesExecutor (tried with CeleryExecutors and exact same problem was observed).

When a Cosmos DbtTaskGroup task fails and is retried, the logs for the failed attempts are not uploaded to S3 (remote logging enabled).
Only the logs from the last retry are available in the Airflow UI and on S3.
This does not happen with standard Airflow operators (e.g. BashOperator), only with Cosmos DbtTaskGroup. What is weird is that if we choose to put no retry at all, we have the logs the first time. But if we choose to put one retry on the airflow dag, we lose the first attempt but we get the second. During the execution of the tasks, we have the logs because airflow is still pulling them directly from the Task pod. But as soon as finished, if the pod have an error, it crashes before being table to upload any log to S3. So logs of the attempt are lost.

The pod running the task is stopped abruptly after failure, before logs can be uploaded.
We see this error in the logs:

airflow.sdk.api.client.ServerResponseError: Server returned errormessage': 'Not Found', 'detail': {'detail': 'Not Found'}}...File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 258, in _from_frame

Full stack trace of the failing pod not succeeding to upload its logs to S3 :

{"timestamp":"2025-10-07T09:37:34.134606Z","level":"warning","event":"The `airflow.models.baseoperator.BaseOperator` attribute is deprecated. Please use `'airflow.sdk.bases.operator.BaseOperator'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:34.135126Z","level":"warning","event":"The `airflow.hooks.base.BaseHook` attribute is deprecated. Please use `'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.384648Z","level":"warning","event":"section/key [openlineage/namespace] not found in config","logger":"airflow.configuration","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:37.385521Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.387229Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.388410Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.389432Z","level":"warning","event":"The `airflow.hooks.base.BaseHook` attribute is deprecated. Please use `'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.403079Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.408536Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.409877Z","level":"warning","event":"The `airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use `'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.803284Z","level":"warning","event":"The `airflow.hooks.base.BaseHook` attribute is deprecated. Please use `'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:38.058447Z","level":"warning","event":"The `airflow.utils.timezone.utcnow` attribute is deprecated. Please use `'airflow.sdk.timezone.utcnow'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:38.383665Z","level":"info","event":"DAG bundles loaded: dags-folder","logger":"airflow.dag_processing.bundles.manager.DagBundlesManager","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.383986Z","level":"info","event":"Filling up the DagBag from /opt/airflow/dags/test/airflow/dags/simple_dag.py","logger":"airflow.models.dagbag.DagBag","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.389834Z","level":"info","event":"Parsing DAG 'simple_dummy_dag' (caller=/opt/airflow/dags/test/airflow/dags/simple_dag.py, env_root=/opt/airflow/dags/test/airflow)","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.389921Z","level":"info","event":"Selected environment: staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.390126Z","level":"info","event":"Env file candidate: /opt/airflow/dags/test/airflow/.env.staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.390578Z","level":"info","event":"Loaded environment variables from /opt/airflow/dags/test/airflow/.env.staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.395721Z","level":"info","event":"Trying to parse the dbt project using dbt ls cache cosmos_cache__simple_dummy_dag__dbt_task_group...","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.420706Z","level":"info","event":"Cosmos performance: time to calculate cache identifier cosmos_cache__simple_dummy_dag__dbt_task_group for current version: 0.0005148248746991158","logger":"cosmos.cache","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.420837Z","level":"info","event":"Cosmos performance [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: The cache size for cosmos_cache__simple_dummy_dag__dbt_task_group is 20132","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421382Z","level":"info","event":"Cosmos performance: Cache hit for cosmos_cache__simple_dummy_dag__dbt_task_group - 116fbfec9b8ac0541dc22732080e7505,8503c37d87e09fa9e1f60b8912e6c0c3","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421689Z","level":"info","event":"Total nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421747Z","level":"info","event":"Total filtered nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421814Z","level":"info","event":"Cosmos performance (simple_dummy_dag__dbt_task_group) - [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.0262s to parse the dbt project for DAG using LoadMode.DBT_LS_CACHE","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.429799Z","level":"info","event":"Cosmos performance (simple_dummy_dag__dbt_task_group) - [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.00796s to build the Airflow DAG.","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.430664Z","level":"info","event":"Trying to parse the dbt project using dbt ls cache cosmos_cache__simple_dummy_dag__minimal_dbt_test...","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440241Z","level":"info","event":"Cosmos performance: time to calculate cache identifier cosmos_cache__simple_dummy_dag__minimal_dbt_test for current version: 0.00046902894973754883","logger":"cosmos.cache","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440335Z","level":"info","event":"Cosmos performance [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: The cache size for cosmos_cache__simple_dummy_dag__minimal_dbt_test is 20132","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440855Z","level":"info","event":"Cosmos performance: Cache hit for cosmos_cache__simple_dummy_dag__minimal_dbt_test - 116fbfec9b8ac0541dc22732080e7505,8503c37d87e09fa9e1f60b8912e6c0c3","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440928Z","level":"info","event":"Total nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440971Z","level":"info","event":"Total filtered nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.441026Z","level":"info","event":"Cosmos performance (simple_dummy_dag__minimal_dbt_test) - [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.0104s to parse the dbt project for DAG using LoadMode.DBT_LS_CACHE","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.446371Z","level":"info","event":"Cosmos performance (simple_dummy_dag__minimal_dbt_test) - [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.00532s to build the Airflow DAG.","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.447464Z","level":"warning","event":"The `airflow.hooks.base.BaseHook` attribute is deprecated. Please use `'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447542Z","level":"warning","event":"The `airflow.operators.bash.BashOperator` attribute is deprecated. Please use `'airflow.providers.standard.operators.bash.BashOperator'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447592Z","level":"warning","event":"Pydantic serializer warnings:\n PydanticSerializationUnexpectedValue(Expected `str` - serialized value may not be as expected [input_value={'version': '116fbfec9b8a...p_id': 'dbt_task_group'}, input_type=dict])","category":"UserWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447633Z","level":"warning","event":"Pydantic serializer warnings:\n PydanticSerializationUnexpectedValue(Expected `str` - serialized value may not be as expected [input_value={'version': '116fbfec9b8a...id': 'minimal_dbt_test'}, input_type=dict])","category":"UserWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:44.835755Z","level":"info","event":"dbtRunner is available. Using dbtRunner for invoking dbt.","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.836090Z","level":"info","event":"Cloning project to writable temp directory /tmp/tmptfu0b0kt from /opt/airflow/dags/test/dbt/test","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.838856Z","level":"info","event":"Partial parse is enabled and the latest partial parse file is None","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.839055Z","level":"info","event":"Using user-supplied profiles.yml at /opt/airflow/dags/test/dbt/test/profiles.yml","logger":"cosmos.config","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.840093Z","level":"info","event":"Trying to run dbtRunner with:\n ['deps', '--project-dir', '/tmp/tmptfu0b0kt', '--profiles-dir', '/opt/airflow/dags/test/dbt/test', '--profile', 'test', '--target', 'staging', '--no-static-parser']\n in /tmp/tmptfu0b0kt","logger":"cosmos.dbt.runner","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.895441Z","level":"info","event":"\u001b[0m09:37:44 Running with dbt=1.11.0-b2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.515707Z","level":"info","event":"\u001b[0m09:37:45 Updating lock file in file path: /tmp/tmptfu0b0kt/package-lock.yml","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.523362Z","level":"info","event":"\u001b[0m09:37:45 Installing /opt/airflow/commons/dbt/data_eng_dbt","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.524967Z","level":"info","event":"\u001b[0m09:37:45 Installed from ","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.525726Z","level":"info","event":"\u001b[0m09:37:45 Installing entechlog/dbt_snow_mask","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.765544Z","level":"info","event":"\u001b[0m09:37:45 Installed from version 0.2.7","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.765723Z","level":"info","event":"\u001b[0m09:37:45 Up to date!","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.766406Z","level":"info","event":"\u001b[0m09:37:45 Installing get-select/dbt_snowflake_query_tags","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.930474Z","level":"info","event":"\u001b[0m09:37:45 Installed from version 2.5.1","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.930654Z","level":"info","event":"\u001b[0m09:37:45 Updated version available: 2.5.3","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.931518Z","level":"info","event":"\u001b[0m09:37:45 Installing dbt-labs/dbt_utils","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.129733Z","level":"info","event":"\u001b[0m09:37:46 Installed from version 1.3.1","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.129896Z","level":"info","event":"\u001b[0m09:37:46 Up to date!","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.130716Z","level":"info","event":"\u001b[0m09:37:46","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.131005Z","level":"info","event":"\u001b[0m09:37:46 Updates available for packages: ['get-select/dbt_snowflake_query_tags']","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.131067Z","level":"info","event":"Update your versions in packages.yml, then run dbt deps","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.495431Z","level":"info","event":"Trying to run dbtRunner with:\n ['run', '--models', 'issues', '--project-dir', '/tmp/tmptfu0b0kt', '--profiles-dir', '/opt/airflow/dags/test/dbt/test', '--profile', 'test', '--target', 'staging', '--no-static-parser']\n in /tmp/tmptfu0b0kt","logger":"cosmos.dbt.runner","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:46.500154Z","level":"info","event":"\u001b[0m09:37:46 Running with dbt=1.11.0-b2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.500980Z","level":"info","event":"\u001b[0m09:37:46 [\u001b[33mWARNING\u001b[0m]: Deprecated functionality","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.501046Z","level":"info","event":"Usage of `--models`, `--model`, and `-m` is deprecated in favor of `--select` or","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.501118Z","level":"info","event":"`-s`.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:47.898632Z","level":"info","event":"\u001b[0m09:37:47 Registered adapter: snowflake=1.10.2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:48.270847Z","level":"info","event":"\u001b[0m09:37:48 Unable to do partial parsing because saved manifest not found. Starting full parse.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.739177Z","level":"info","event":"\u001b[0m09:37:49 Found 5 models, 22 data tests, 625 macros","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752263Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752374Z","level":"info","event":"\u001b[0m09:37:49 Concurrency: 16 threads (target='staging')","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752445Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.768383Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.768650Z","level":"info","event":"\u001b[0m09:37:49 Finished running in 0 hours 0 minutes and 0.03 seconds (0.03s).","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769138Z","level":"info","event":"\u001b[0m09:37:49 Encountered an error:","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769208Z","level":"info","event":"Runtime Error","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769273Z","level":"info","event":" Database error while listing schemas in database \"TEST_STAGING\"","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769338Z","level":"info","event":" Database Error","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769402Z","level":"info","event":" ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [])","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769646Z","level":"info","event":"\u001b[0m09:37:49 [\u001b[33mWARNING\u001b[0m][DeprecationsSummary]: Deprecated functionality","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769711Z","level":"info","event":"Summary of encountered deprecations:","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769775Z","level":"info","event":"- ModelParamUsageDeprecation: 1 occurrence","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769857Z","level":"info","event":"To see all deprecation instances instead of just the first occurrence of each,","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769919Z","level":"info","event":"run command again with the `--show-all-deprecations` flag. You may also need to","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769998Z","level":"info","event":"run with `--no-partial-parse` as some deprecations are only encountered during","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.770073Z","level":"info","event":"parsing.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:50.159921Z","level":"error","event":"Task failed with exception","logger":"task","filename":"supervisor.py","lineno":1739,"error_detail":[{"exc_type":"CosmosDbtRunError","exc_value":"dbt invocation did not complete with unhandled error: Runtime Error\n Database error while listing schemas in database \"TEST_STAGING\"\n Database Error\n ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [])","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":920,"name":"run"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1307,"name":"_execute_task"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/base.py","lineno":310,"name":"execute"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":853,"name":"build_and_run_cmd"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":650,"name":"run_command"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":258,"name":"handle_exception_dbt_runner"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/dbt/runner.py","lineno":109,"name":"handle_exception_if_needed"}],"is_group":false,"exceptions":[]}]}
{"timestamp":"2025-10-07T09:37:50.203777Z","level":"error","event":"API server error","status_code":404,"detail":{"detail":"Not Found"},"message":"Not Found","logger":"supervisor","filename":"supervisor.py","lineno":616}
{"timestamp":"2025-10-07T09:37:50.204113Z","level":"error","event":"Top level error","logger":"task","filename":"supervisor.py","lineno":1739,"error_detail":[{"exc_type":"AirflowRuntimeError","exc_value":"API_SERVER_ERROR: {'status_code': 404, 'message': 'Not Found', 'detail': {'detail': 'Not Found'}}","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1452,"name":"main"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1397,"name":"finalize"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py","lineno":207,"name":"send"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py","lineno":271,"name":"_get_response"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py","lineno":258,"name":"_from_frame"}],"is_group":false,"exceptions":[]}]}
{"timestamp":"2025-10-07T09:37:50.244332Z","level":"error","event":"/usr/python/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown","logger":"task.stderr","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:50.244508Z","level":"error","event":" warnings.warn('resource_tracker: There appear to be %d '","logger":"task.stderr","filename":"supervisor.py","lineno":1754}
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 125, in
main()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 121, in main
execute_workload(workload)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 66, in execute_workload
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1887, in supervise
exit_code = process.wait()
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 988, in wait
self.update_task_state_if_needed()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1002, in update_task_state_if_needed
self.client.task_instances.finish(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 221, in finish
self.client.patch(f"task-instances/{id}/state", content=body.model_dump_json())
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 861, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
hook(response)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 181, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 171, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Server returned error

### What you think should happen instead?

Even if the task fail, for each retries, I should always get the logs available in the UI.

### How to reproduce

1. Create a DbtTaskGroup with a failing dbt model and retries > 0
2. Set the tasks as KubernetesExecutors
3. Choose S3 for remote logging with airflow
4. Trigger the DAG
5. Observe that only the last retry's logs are available in S3/UI

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Reproduce a failed retry for a Cosmos DbtTaskGroup with S3 remote logging and compare it with a BashOperator. Start by inspecting airflow/sdk/execution_time/comms.py around line 258 and the failing supervisor.py trace. Done means logs from every failed attempt are uploaded to S3 and remain available in the Airflow UI.

Written by the indexing model from the issue text.