airbytehq / airbytehq/airbyte

Socket-based syncs blocked by AKS ContainerNoPrivilegeEscalation Policy

Aperta
#83,782 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Helm Chart Version

2.0.17

### What step the error happened?

During the Sync

### Relevant information

Following an update of the Databricks destination to 4.0.1, connections from sources supporting `SOCKET` transport began failing with this error during the sync step:

```
ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0.
message: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0.
stackTrace: [Ljava.lang.StackTraceElement;@5e61742d
```

Sources that only supported `STDOI` continued to work.

### Debugging

The failure was traced to an AKS gatekeeper denial for a ContainerNoPrivilegeEscalation policy violation. I was able to repro this failure using a security context based on [the `securityContext` created](https://github.com/airbytehq/airbyte-platform/blob/2d96861799e85bec7ddc9a60cccd93866bd549e2/airbyte-workload-launcher/src/main/kotlin/pods/factories/ReplicationPodFactory.kt#L125-L126) when the `SOCKET` transport is selected:

```
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
```

When tested against the AKS gatekeeper:

```
PS > kubectl create --dry-run=server -f "socket-dryrun-pod.yaml" 2>&1
Error from server (Forbidden): error when creating "socket-dryrun-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: destination
[azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: init
[azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: orchestrator
[azurepolicy-k8sazurev3noprivilegeescalatio-f398463faa240def6d38] Privilege escalation container is not allowed: source
```

The ContainerNoPrivilegeEscalation policy (https://github.com/Azure/azure-policy/blob/f600c952718e8d2fc1c4fe92c516100d2ad85999/built-in-policies/policyDefinitions/Kubernetes/ContainerNoPrivilegeEscalation.json#L169) is set to block `securityContext` configs that use any value other than `false` if they set a `runAsUser` value other than zero:

```
PS > kubectl get constrainttemplate k8sazurev3noprivilegeescalation -o jsonpath='{.spec.targets[0].rego}'
package k8sazurev3noprivilegeescalation

import data.lib.exclude_container_image.is_excluded

# The following scenerios will be viewed as violations. Windows containers are not supported
## allowPrivilegeEscalation=true
## runAsUser != 0 + missing allowPrivilegeEscalation=false

violation[{"msg": msg, "details": {}}] {
not input.review.object.spec.nodeSelector["kubernetes.io/os"] == "windows"
c := input_containers[_]
not input_container_excluded(c.name)
not is_excluded(c)
input_allow_privilege_escalation(c)
msg := sprintf("Privilege escalation container is not allowed: %v", [c.name])
}

input_allow_privilege_escalation(c) {
c.securityContext.allowPrivilegeEscalation == true
}
input_allow_privilege_escalation(c) {
not c.securityContext.allowPrivilegeEscalation == false
c.securityContext.runAsUser != 0
}
input_allow_privilege_escalation(c) {
not c.securityContext.allowPrivilegeEscalation == false
not c.securityContext.runAsUser
input.review.object.spec.securityContext.runAsUser != 0
}
input_containers[c] {
c := input.review.object.spec.containers[_]
}
input_containers[c] {
c := input.review.object.spec.initContainers[_]
}
input_containers[c] {
c := input.review.object.spec.ephemeralContainers[_]
}
input_container_excluded(field) {
field == input.parameters.excludedContainers[_]
}
```

### Solutions

Given an updated `securityContext` value with `allowPrivilegeEscalation: false`, the gatekeeper dry-run permitted the pod creation. This is consistent with the definition for [`WorkloadSecurityContextProvider.baseContainerSecurityContext()`](https://github.com/airbytehq/airbyte-platform/blob/v1.8.0/airbyte-workload-launcher/src/main/kotlin/context/WorkloadSecurityContextProvider.kt#L104-L114):

```
private fun baseContainerSecurityContext(
user: Long,
group: Long,
): SecurityContextBuilder =
SecurityContextBuilder()
.withRunAsUser(user)
.withRunAsGroup(group)
.withAllowPrivilegeEscalation(false)
.withReadOnlyRootFilesystem(false)
.withCapabilities(CapabilitiesBuilder().addAllToDrop(DEFAULT_CAPABILITIES).build())
}
```

The current workaround is to use an AKS cluster mutation policy to inject `allowPrivilegeEscalation: false`, but ideally Airbyte should work out of the box with the ContainerNoPrivilegeEscalation policy in effect?

The issue reported at https://github.com/airbytehq/airbyte/issues/77613 is partially related, but in that case the `securityContext` is reported as an empty `{}` which is what we're seeing for `STDIO` sources without a gatekeeper failure.

### Relevant log output

```shell
2026-08-06 14:18:02 platform INFO APPLY Stage: BUILD — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO APPLY Stage: CLAIM — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO Claimed: true for workload c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync via API in dataplane AUTO3fbd01c3-f8e0-4107-9f4a-dd14270550a1 (3cb4b628-3345-4590-bd66-ddf751089fd8)
2026-08-06 14:18:02 platform INFO APPLY Stage: LOAD_SHED — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO No pod found running for workload c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync
2026-08-06 14:18:02 platform INFO APPLY Stage: MUTEX — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO Mutex key: c20f60e7-c0e9-440f-8795-73d841838c28 specified for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync. Attempting to delete existing pods...
2026-08-06 14:18:02 platform INFO Mutex key: c20f60e7-c0e9-440f-8795-73d841838c28 specified for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync found no existing pods. Continuing...
2026-08-06 14:18:02 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO APPLY Stage: LAUNCH — (workloadId=c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync)
2026-08-06 14:18:02 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=1024Mi, cpu=1}, requests={memory=1024Mi, cpu=1}, additionalProperties={})
2026-08-06 14:18:02 platform INFO Launching replication pod: replication-job-15469-attempt-0 (selectors = {workload_type=jobs}) with containers:
2026-08-06 14:18:02 platform INFO [source] image: airbyte/source-mysql:3.51.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=1}, requests={memory=2Gi, cpu=0.5}, additionalProperties={})
2026-08-06 14:18:02 platform INFO [destination] image: airbyte/destination-databricks:4.0.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=1}, requests={memory=2Gi, cpu=0.5}, additionalProperties={})
2026-08-06 14:18:02 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=1024Mi, cpu=1}, requests={memory=1024Mi, cpu=1}, additionalProperties={})
2026-08-06 14:18:02 platform ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0.
message: io.airbyte.workers.exception.KubeClientException: Failed to create pod replication-job-15469-attempt-0.
stackTrace: [Ljava.lang.StackTraceElement;@5e61742d
2026-08-06 14:18:02 platform INFO Attempting to update workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync to FAILED.
2026-08-06 14:18:02 platform INFO Pipeline aborted after error for workload: c20f60e7-c0e9-440f-8795-73d841838c28_15469_0_sync.
2026-08-06 14:18:03 platform INFO
----- START POST REPLICATION OPERATIONS -----

2026-08-06 14:18:03 platform INFO No post-replication operation(s) to perform.
2026-08-06 14:18:03 platform INFO
----- END POST REPLICATION OPERATIONS -----
```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12122

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.