🐛 [Bug Report] Airbyte 2.1.0 -> MySQL CDC sync jobs fail repeatedly when GKE cluster autoscaler evicts Temporal/Workload API Server pods mid-flight

### Topic

_No response_

### Relevant information

*Environment*
- Airbyte: `2.1.0` (self-hosted on Kubernetes / GKE)
- Connector: MySQL Source (CDC / Debezium binlog)
- Deployment: GKE with cluster autoscaler enabled

---

*Summary*

A MySQL CDC replication job failed permanently after *15 attempts over ~2.5 hours*, syncing *0 records*. The root cause was not a connector bug: the GKE cluster autoscaler was evicting the `temporal` and `workload-api-server` pods multiple times during the job run, causing each replication pod to be killed mid-flight and the `ConnectionManagerWorkflow` to lose its sticky worker.

---

*Step 1 — Temporal loses its sticky worker at job start*

```
[05:30:02Z] level=error
error="sticky worker unavailable, please use original task queue."
wf-id="connection_manager_"
operation="AddWorkflowTask"
```

*Step 2 — Workload launcher loop: each new attempt kills the previous pod via MUTEX*

```
[05:30:14Z] APPLY Stage: CHECK_STATUS — No pod found running
[05:30:15Z] APPLY Stage: MUTEX — Attempting to delete existing pods...
[05:30:15Z] APPLY Stage: LAUNCH — (pod created)

-- ~4 minutes later --

[05:38:27Z] APPLY Stage: MUTEX — Existing pods for mutex key deleted.
[05:38:27Z] APPLY Stage: LAUNCH — (new pod created)
```

This loop repeated from attempt `91_0` through `91_14`.

*Step 3 — Inside the replication pod: CDC phase succeeds, source crashes during JDBC stream read*

```
# CDC phase: OK
[06:00:20Z] INFO: Connected to :3306 at binlog pos dbabc84.038207/77664758
[06:00:41Z] INFO: Stopped reading binlog after ~42000 events
[06:00:41Z] INFO: Connection gracefully closed

# JDBC stream phase: crashes immediately after
[06:00:59Z] WARN: Unhandled message: Undefined ← socket closed, source process killed
[06:00:59Z] WARN: Unhandled message: Undefined
[06:00:59Z] WARN: Unhandled message: Undefined
# log ends abruptly
```

*Step 4 — Final failure report (attempt 14)*

```
[07:00:49Z] ERROR - io.airbyte.cdk.TransientErrorException:
Input was fully read, but some streams did not receive a terminal
stream status message. Streams without status:
[porp.f_disputes_movements_intermediary,
porp.ds_mts_globe_pending_bookings,
porp.fp_events_etl, porp.fp_transcoding,
porp.f_disputes_movements, ...]

[07:00:59Z] ERROR - Source process exited with non-zero exit code 3

[07:59:27Z] ERROR - Failing job: 91, reason: Job failed after too many retries
```

*Step 5 — Even reporting the failure back to Temporal fails (Temporal itself was evicted)*

```
[08:00:02Z] WARN - Failure during reporting of activity result to the server.
ActivityType=JobFailure, WorkflowId=connection_manager_

[08:09:27Z] ERROR - [ACTIVITY-FAILURE] Connection failed to run activity.
(JobFailureInput). Workflow will be restarted after a delay of PT10M.
```

Repeated 3 more times (08:29, 08:50...).

*Step 6 — Root cause confirmed via kubectl*

```
kubectl get events -n airbyte-experimental --sort-by='.lastTimestamp'

Normal ScaleDown pod/airbyte-experimental-temporal-...-6fpkw
deleting pod for node scale down

Normal ScaleDown pod/airbyte-experimental-workload-api-server-...-9vgz7
deleting pod for node scale down

Normal ScaleDown pod/airbyte-experimental-temporal-...-5fdng
deleting pod for node scale down
```

Three separate scale-down events in ~25 minutes, each evicting Temporal and/or the Workload API Server.

---

*Workaround applied*

```
kubectl patch deployment airbyte-experimental-temporal \
-n airbyte-experimental \
--type=json \
-p='[{"op":"add","path":"/spec/template/metadata/annotations/cluster-autoscaler.kubernetes.io~1safe-to-evict","value":"false"}]'

kubectl patch deployment airbyte-experimental-workload-api-server \
-n airbyte-experimental \
--type=json \
-p='[{"op":"add","path":"/spec/template/metadata/annotations/cluster-autoscaler.kubernetes.io~1safe-to-evict","value":"false"}]'
```

---

*Questions / suggestions for the Airbyte team*

- Should `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` be added by default in the Helm chart for `temporal`, `workload-api-server`, and `server`? These hold live workflow state and losing them silently causes cascading failures that are very hard to diagnose.
- Retries caused by infrastructure eviction shouldn't count toward `maxSyncJobAttempts`. Is there a way to detect pod eviction as a distinct failure class?
- The error shown in the UI (`exit code 3` / `TransientErrorException`) is the symptom, not the cause. Surfacing the eviction reason in the job failure message would save significant debugging time.
- When Temporal is down, the `JobFailure` activity retries indefinitely. Is there a dead-letter path or circuit-breaker for this?

Happy to share additional log excerpts if useful. Thanks! 🙏

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12130