airbytehq / airbytehq/airbyte

[helm] Cannot start replication pods when using a service account not named `airbyte-admin` (Airbyte OSS 1.3.1)

Ouverte
#49,938 12 commentaires 5 réactions 0 personnes assignées Voir sur GitHub
area/platform community team/deployments type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.4k
Merge moyen
5 h
PR mergées (30 j)
671

Description

### Helm Chart Version

1.3.1

### What step the error happened?

During the Sync

### Relevant information

## Summary

After upgrading Airbyte to 1.3.1 (OSS), when workload-launcher starts a replication pod, it does a PATCH call to pods it is trying to launch, however the PATCH call seems to try to use `airbyte-admin` as its ServiceAccount even though a different service account name is specified in the `values.yaml` (the pod being started has uses the correct service account and has the `JOB_KUBE_SERVICEACCOUNT` from the values file).

## Step to reproduce

- Airbyte 1.3.1 (OSS) deployed using helm chart
- Use a ServiceAccount name that's different from `airbyte-admin` but has the necessary role bindings
- Start a sync for a connection
- When the pod start you should see the following exception in the `workload-launcher` pod logs:

```
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod postgres-discover-.
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:227)
at io.airbyte.workload.launcher.pods.KubePodClient.launchDiscover(KubePodClient.kt:185)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:50)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found.
```

Let me know if you need any further information, I'll be happy to help.

## Additional information

- Kurbenetes cluster is running on EKS

## Note

The issue is not happening on 1.2.0.

## Workaround

As a workaround I created another service account named `airbyte-admin` having the same role binding as the role we normally use.
Then workload-launcher will be able to start the replication pod.

### Relevant log output

```shell
2024-12-19 09:32:02,134 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,143 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,144 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: _check. Continuing...
2024-12-19 09:32:02,144 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,147 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.f.InitContainerFactory(create):42 - [initContainer] image: airbyte/workload-init-container:1.3.1 resources: ResourceRequirements(claims=], limits={}, requests={}, additionalProperties={})
at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.apply(Unknown Source)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:24)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:193)
at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:84)
at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4642)
at reactor.core.publisher.Mono.subscribe(Mono.java:4403)
at io.airbyte.workload.launcher.pipeline.LaunchPipeline.accept(LaunchPipeline.kt:50)
at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:28)
at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:12)
at io.airbyte.commons.temporal.queue.QueueActivityImpl.consume(Internal.kt:87)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
at io.temporal.common.interceptors.ActivityInboundCallsInterceptorBase.execute(ActivityInboundCallsInterceptorBase.java:39)
at io.temporal.opentracing.internal.OpenTracingActivityInboundCallsInterceptor.execute(OpenTracingActivityInboundCallsInterceptor.java:78)
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod .
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:227)
at io.airbyte.workload.launcher.pods.KubePodClient.launchCheck(KubePodClient.kt:167)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:49)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=], group=null, kind=pods, name=, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:251)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1179)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:98)
at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:56)
at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:51)
at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand$lambda$2(KubePodLauncher.kt:322)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand(KubePodLauncher.kt:322)
at io.airbyte.workload.launcher.pods.KubePodLauncher.create(KubePodLauncher.kt:51)
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:224)
... 57 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=], group=null, kind=pods, name=, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
... 3 common frames omitted

2024-12-19 09:32:02,543 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.c.WorkloadApiClient(updateStatusToFailed):54 - Attempting to update workload: _check to FAILED.
2024-12-19 09:32:02,562 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.h.FailureHandler(apply):62 - Pipeline aborted after error for workload: _check.
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.