airbytehq / airbytehq/airbyte

[helm] Cannot start replication pods when using a service account not named `airbyte-admin` (Airbyte OSS 1.3.1)

Đang mở
#49,938 12 bình luận 5 reaction 0 người được giao Xem trên GitHub
area/platform community team/deployments type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.4k
Merge trung bình
5 giờ
Pull request đã merge (30 ngày)
671

Mô tả

### Helm Chart Version

1.3.1

### What step the error happened?

During the Sync

### Relevant information

## Summary

After upgrading Airbyte to 1.3.1 (OSS), when workload-launcher starts a replication pod, it does a PATCH call to pods it is trying to launch, however the PATCH call seems to try to use `airbyte-admin` as its ServiceAccount even though a different service account name is specified in the `values.yaml` (the pod being started has uses the correct service account and has the `JOB_KUBE_SERVICEACCOUNT` from the values file).

## Step to reproduce

- Airbyte 1.3.1 (OSS) deployed using helm chart
- Use a ServiceAccount name that's different from `airbyte-admin` but has the necessary role bindings
- Start a sync for a connection
- When the pod start you should see the following exception in the `workload-launcher` pod logs:

```
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod postgres-discover-.
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:227)
at io.airbyte.workload.launcher.pods.KubePodClient.launchDiscover(KubePodClient.kt:185)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:50)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found.
```

Let me know if you need any further information, I'll be happy to help.

## Additional information

- Kurbenetes cluster is running on EKS

## Note

The issue is not happening on 1.2.0.

## Workaround

As a workaround I created another service account named `airbyte-admin` having the same role binding as the role we normally use.
Then workload-launcher will be able to start the replication pod.

### Relevant log output

```shell
2024-12-19 09:32:02,134 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: BUILD — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,143 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: MUTEX — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,144 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.EnforceMutexStage(applyStage):50 - No mutex key specified for workload: _check. Continuing...
2024-12-19 09:32:02,144 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.s.m.Stage(apply):39 - APPLY Stage: LAUNCH — (workloadId = _check) — (dataplaneId = local)
2024-12-19 09:32:02,147 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.f.InitContainerFactory(create):42 - [initContainer] image: airbyte/workload-init-container:1.3.1 resources: ResourceRequirements(claims=], limits={}, requests={}, additionalProperties={})
at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.apply(Unknown Source)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:24)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:193)
at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)
at reactor.core.scheduler.ImmediateScheduler$ImmediateSchedulerWorker.schedule(ImmediateScheduler.java:84)
at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55)
at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4642)
at reactor.core.publisher.Mono.subscribe(Mono.java:4403)
at io.airbyte.workload.launcher.pipeline.LaunchPipeline.accept(LaunchPipeline.kt:50)
at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:28)
at io.airbyte.workload.launcher.pipeline.consumer.LauncherMessageConsumer.consume(LauncherMessageConsumer.kt:12)
at io.airbyte.commons.temporal.queue.QueueActivityImpl.consume(Internal.kt:87)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
at io.temporal.common.interceptors.ActivityInboundCallsInterceptorBase.execute(ActivityInboundCallsInterceptorBase.java:39)
at io.temporal.opentracing.internal.OpenTracingActivityInboundCallsInterceptor.execute(OpenTracingActivityInboundCallsInterceptor.java:78)
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod .
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:227)
at io.airbyte.workload.launcher.pods.KubePodClient.launchCheck(KubePodClient.kt:167)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:49)
at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:24)
at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:42)
... 53 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=], group=null, kind=pods, name=, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236)
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:251)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1179)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:98)
at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:56)
at io.airbyte.workload.launcher.pods.KubePodLauncher$create$1.invoke(KubePodLauncher.kt:51)
at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand$lambda$2(KubePodLauncher.kt:322)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand(KubePodLauncher.kt:322)
at io.airbyte.workload.launcher.pods.KubePodLauncher.create(KubePodLauncher.kt:51)
at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar(KubePodClient.kt:224)
... 57 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://:/api/v1/namespaces/airbyte/pods/?fieldManager=fabric8. Message: pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=], group=null, kind=pods, name=, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "" is forbidden: error looking up service account airbyte/airbyte-admin: serviceaccount "airbyte-admin" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
... 3 common frames omitted

2024-12-19 09:32:02,543 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.c.WorkloadApiClient(updateStatusToFailed):54 - Attempting to update workload: _check to FAILED.
2024-12-19 09:32:02,562 [Activity Executor taskQueue="workload_default", namespace="default": 2] INFO i.a.w.l.p.h.FailureHandler(apply):62 - Pipeline aborted after error for workload: _check.
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.