Connector-Sidecar Cannot Authenticate to Workload API Server in v2.0.1 OSS

### Helm Chart Version

2.0.19

### What step the error happened?

Other

### Relevant information

# Connector-Sidecar Cannot Authenticate to Workload API Server in v2.0.1 OSS

## Environment
- **Airbyte Version**: 2.0.1 (Helm Chart 2.0.19)
- **Edition**: Community (OSS)
- **Deployment**: Kubernetes (AWS EKS 1.31)
- **Storage**: S3 with IAM instance profile authentication
- **Database**: External RDS PostgreSQL

## Problem
The connector-sidecar containers created by workload-launcher do not receive `WORKLOAD_API_AUTH_HEADER` and `WORKLOAD_API_BEARER_TOKEN` environment variables, causing 401 Unauthorized errors when trying to write results to the workload API server. This prevents any source connection tests from completing in the UI (504 Gateway Timeout), even though the actual MySQL connections work perfectly.

## What's Working
✅ MySQL source connector successfully connects to database (verified directly)
✅ Network/VPC connectivity is correct
✅ DNS resolution working
✅ S3 storage properly configured with all required buckets
✅ Auth tokens configured on all components
✅ All Airbyte pods running and healthy

## Steps to Reproduce

1. Deploy Airbyte v2.0.1 with Helm chart 2.0.19
2. Configure workload API authentication as recommended:
```yaml
workloadApiServer:
extraEnv:
- name: WORKLOAD_API_AUTH_HEADER
value: "Authorization"
- name: WORKLOAD_API_BEARER_TOKEN
value: "[generated-token]"

worker:
extraEnv:
- name: WORKLOAD_API_AUTH_HEADER
value: "Authorization"
- name: WORKLOAD_API_BEARER_TOKEN
value: "[same-token]"

workloadLauncher:
extraEnv:
- name: WORKLOAD_API_AUTH_HEADER
value: "Authorization"
- name: WORKLOAD_API_BEARER_TOKEN
value: "[same-token]"
```
3. Try to add any source (e.g., MySQL) via the UI
4. Observe 504 Gateway Timeout in UI
5. Check connector-sidecar logs: shows repeated HTTP 500 errors trying to POST to output endpoint

## Root Cause Analysis

### Step 1: Missing Endpoint (FIXED)
Initially, the endpoint `/api/v1/workload/{id}/output` returned 404 Not Found.

**Fix:** Set `API_AUTHORIZATION_ENABLED=true` in ConfigMap `airbyte-airbyte-env`.

**Result:** Endpoint now returns 401 Unauthorized (endpoint exists, but auth missing).

### Step 2: Connector-Sidecar Missing Auth Token (CURRENT ISSUE)
The connector-sidecar containers don't have the auth token, even though it's configured on workload-launcher:

**workload-launcher deployment has:**
```bash
kubectl get deployment airbyte-workload-launcher -n airbyte-v2 -o json | jq -r '.spec.template.spec.containers[0].env[] | select(.name | contains("WORKLOAD")) | "\(.name): \(.value)"'
```
Output:
```
WORKLOAD_API_AUTH_HEADER: Authorization
WORKLOAD_API_BEARER_TOKEN: [generated-token]
```

**But connector-sidecar pods DON'T have it:**
```bash
kubectl get pod source-mysql-check-XXX -n airbyte-v2 -o json | jq -r '.spec.containers[] | select(.name=="connector-sidecar") | .env[] | select(.name | contains("WORKLOAD")) | "\(.name): \(.value)"'
```
Output:
```
WORKLOAD_API_READ_TIMEOUT_SECONDS: 300
WORKLOAD_API_RETRY_DELAY_SECONDS: 2
WORKLOAD_API_CONNECT_TIMEOUT_SECONDS: 30
WORKLOAD_API_MAX_RETRIES: 5
WORKLOAD_API_HOST: http://airbyte-workload-api-server-svc.airbyte-v2:8001
```

**Missing:** `WORKLOAD_API_AUTH_HEADER` and `WORKLOAD_API_BEARER_TOKEN`

### Step 3: MySQL Connection Works
From the connector-sidecar logs, the MySQL connector successfully completes:
```
INFO main i.a.c.c.CheckOperation(execute):51 - Config check completed successfully.
INFO main i.a.c.ConnectorWatcher(saveConnectorOutput):186 - Writing output of 435bb9a5-7887-4809-aa58-28c27df0d7ad_458f552f-e825-4e2c-90d5-0d53bcb2b07c_0_check to the doc store
WARN i.a.a.c.c.ClientConfigurationSupport - Retry attempt 1 of 5. Last response: null
```

The connector succeeds, but fails to write results because the connector-sidecar doesn't have auth credentials.

## Configuration

### Helm Values (helm/values.yaml)
```yaml
global:
database:
type: external
secretName: "airbyte-config-secrets"
host: "airbyte-cluster-db.cnck88s0iflt.us-east-2.rds.amazonaws.com"
port: "5432"
name: "airbyte"
storage:
type: "s3"
secretName: "airbyte-config-secrets"
bucket:
log: "stellarfi-airbyte"
state: "stellarfi-airbyte"
workloadOutput: "stellarfi-airbyte"
auditLogging: "stellarfi-airbyte"
activityPayload: "stellarfi-airbyte"
s3:
region: "us-east-2"
authenticationType: "instanceProfile"

workloadApiServer:
extraEnv:
- name: AWS_DEFAULT_REGION
value: "us-east-2"
- name: AWS_REGION
value: "us-east-2"
- name: STORAGE_REGION
value: "us-east-2"
- name: S3_REGION
value: "us-east-2"
- name: WORKLOAD_API_AUTH_HEADER
value: "Authorization"
- name: WORKLOAD_API_BEARER_TOKEN
value: "[generated-token]"
```

### ConfigMap Environment Variables (Verified)
```
WORKLOAD_API_SERVER_ENABLED=true
WORKLOAD_API_HOST=http://airbyte-workload-api-server-svc.airbyte-v2:8001
STORAGE_TYPE=s3
STORAGE_BUCKET_LOG=stellarfi-airbyte
STORAGE_BUCKET_STATE=stellarfi-airbyte
STORAGE_BUCKET_WORKLOAD_OUTPUT=stellarfi-airbyte
STORAGE_BUCKET_AUDIT_LOGGING=stellarfi-airbyte
STORAGE_BUCKET_ACTIVITY_PAYLOAD=stellarfi-airbyte
```

### Workload API Server Pod Environment (Verified)
```bash
$ kubectl exec -n airbyte-v2 airbyte-workload-api-server-xxx -- env | grep WORKLOAD
WORKLOAD_API_SERVER_ENABLED=true
WORKLOAD_API_HOST=http://airbyte-workload-api-server-svc.airbyte-v2:8001
WORKLOAD_API_AUTH_HEADER=Authorization
WORKLOAD_API_BEARER_TOKEN=[generated-token]
```

## Logs

### Workload API Server Startup
```
2025-12-08 18:30:43,653 [main] INFO i.m.r.Micronaut(start):101 - Startup completed in 8893ms. Server Running: http://airbyte-workload-api-server-77798fdbcc-n2twj:8007
```

Server starts successfully with no errors.

### Connector Sidecar Logs (source-mysql-check pod)
```
2025-12-08 18:27:34,764 [main] ERROR i.a.a.c.c.ClientConfigurationSupport - Failed to call unknown. Last response: null
java.io.IOException: HTTP error: 500 Internal Server Error
at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16)
...
at io.airbyte.api.client.generated.WorkloadOutputApi.writeWorkloadOutput(WorkloadOutputApi.kt:62)
at io.airbyte.workers.workload.WorkloadOutputWriter.writeOutputThroughServer(WorkloadOutputWriter.kt:82)
at io.airbyte.workers.workload.WorkloadOutputWriter.write(WorkloadOutputWriter.kt:65)
at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:187)
...

2025-12-08 18:27:34,766 [main] ERROR i.a.c.ConnectorWatcher(handleException):200 - Error performing operation: io.airbyte.workers.workload.exception.DocStoreAccessException
io.airbyte.workers.workload.exception.DocStoreAccessException: Unable to write output for 435bb9a5-7887-4809-aa58-28c27df0d7ad_da898f67-0848-4556-9c1c-3c4792bdba05_0_check
```

The connector-sidecar retries 5 times, all failing with HTTP 500/404.

## Expected Behavior
The `POST /api/v1/workload/{id}/output` endpoint should be available and accept connector output results.

## Actual Behavior
The endpoint returns 404 Not Found, preventing any source connection tests from completing.

## Additional Context

**Confirmed by Airbyte Slack Support Bot (Dec 8, 2024):**
> "There is no documented Helm value in chart v2 that injects WORKLOAD_API_AUTH_HEADER / WORKLOAD_API_BEARER_TOKEN into the connector-sidecar containers"
>
> "In working examples, WORKLOAD_API_BEARER_TOKEN for sidecars comes from a secret wired into the platform templates, not from Helm-user-configurable extraEnv"
>
> "The behavior you're seeing—launcher has the vars, sidecar does not, causing 401/500 on writeWorkloadOutput—is consistent with an omission in the job-pod/sidecar env propagation logic"
>
> "This is very likely a bug or regression rather than a config oversight"

**Related Issues:**
- Users have reported similar issues with proxy env vars not propagating to connector-sidecar containers
- Other users with 401/502 errors only needed to set tokens on worker and workload-api-server (which we have done)
- This appears to be a regression in v2.0.1 OSS where the platform doesn't wire secrets into connector-sidecar containers

## Question
How do we configure the Helm chart so that workload-launcher passes `WORKLOAD_API_AUTH_HEADER` and `WORKLOAD_API_BEARER_TOKEN` to the connector-sidecar containers it creates?

We've tried:
1. ✅ Setting them in `workloadLauncher.extraEnv` - workload-launcher has them, but doesn't pass them to connector-sidecar
2. ✅ Adding them to global ConfigMap `airbyte-airbyte-env` - connector-sidecar still doesn't get them
3. ❌ No `sidecar.extraEnv` or similar configuration found in Helm chart

Is there a missing Helm value or configuration option to inject these env vars into dynamically created connector-sidecar containers?

## Deployment Command
```bash
helm upgrade --install airbyte airbyte-v2/airbyte \
--namespace airbyte-v2 \
--create-namespace \
--version 2.0.19 \
--values helm/values.yaml \
--wait
```

### Relevant log output

```shell

```