Airbyte Pod Intermittently Losing IRSA Permissions and Falling Back to Node IAM Role

### Helm Chart Version

1.3.1

### What step the error happened?

Other

### Relevant information

I'm encountering an intermittent issue with an Airbyte pod running on EKS, which is configured to assume an AWS IAM role using IRSA. Every few days, the pod appears to lose the service account permissions and instead starts using the IAM role associated with the underlying EKS node. This results in permission-denied errors when attempting to access an S3 bucket.

Expected Behavior:
The Airbyte pod should consistently assume the IAM role associated with its service account via IRSA and retain the expected permissions throughout its lifecycle.

### Relevant log output

```shell
2025-02-11 13:49:44.286
INFO i.a.c.s.h.SchedulerHandler(createJob):602 - Found the following streams to reset for connection : []
2025-02-11 13:49:44.320
INFO i.a.p.j.DefaultJobPersistence(enqueueJob):582 - Enqueuing pending job for scope:
2025-02-11 13:49:44.324
INFO i.a.c.s.h.SchedulerHandler(createJob):650 - New job created, with id:
2025-02-11 13:49:44.733
WARN i.a.c.j.JsonSchemas(traverseJsonSchemaInternal):200 - The object is a properties key or a combo keyword. The traversal is silently stopped. Current schema: {"type":"object","airbyte_hidden":true,"additionalProperties":true}
2025-02-11 13:49:44.787
WARN i.a.c.j.JsonSchemas(traverseJsonSchemaInternal):200 - The object is a properties key or a combo keyword. The traversal is silently stopped. Current schema: {"type":"object","airbyte_hidden":true,"additionalProperties":true}
2025-02-11 13:50:04.777
ERROR i.a.s.a.ApiHelper(execute):46 - Unexpected Exception
software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts:::assumed-role// is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::" because no identity-based policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: , Extended Request ID: )
2025-02-11 13:50:04.777
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
2025-02-11 13:50:04.777
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
2025-02-11 13:50:04.777
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
2025-02-11 13:50:04.777
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
2025-02-11 13:50:04.777
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:93)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
2025-02-11 13:50:04.777
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
2025-02-11 13:50:04.777
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:7327)
2025-02-11 13:50:04.777
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:154)
2025-02-11 13:50:04.777
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:145)
2025-02-11 13:50:04.777
at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
2025-02-11 13:50:04.777
at io.airbyte.commons.storage.AbstractS3StorageClient.list(StorageClient.kt:571)
2025-02-11 13:50:04.777
at io.airbyte.commons.logging.LogClient.getLogs(LogClient.kt:95)
2025-02-11 13:50:04.777
at io.airbyte.commons.logging.LogClientManager.getLogs(LogClientManager.kt:50)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.converters.JobConverter.getAttemptLogs(JobConverter.java:306)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:334)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:329)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.handlers.SchedulerHandler.retrieveDiscoveredSchema(SchedulerHandler.java:567)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.handlers.SchedulerHandler.discoverAndGloballyDisable(SchedulerHandler.java:400)
2025-02-11 13:50:04.777
at io.airbyte.commons.server.handlers.SchedulerHandler.discoverSchemaForSourceFromSourceId(SchedulerHandler.java:365)
2025-02-11 13:50:04.777
at io.airbyte.server.apis.SourceApiController.lambda$discoverSchemaForSource$6(SourceApiController.java:114)
2025-02-11 13:50:04.777
at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:31)
2025-02-11 13:50:04.777
at io.airbyte.server.apis.SourceApiController.discoverSchemaForSource(SourceApiController.java:114)
2025-02-11 13:50:04.777
at io.airbyte.server.apis.$SourceApiController$Definition$Exec.dispatch(Unknown Source)
2025-02-11 13:50:04.777
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
2025-02-11 13:50:04.777
at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4354)
2025-02-11 13:50:04.777
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
2025-02-11 13:50:04.777
at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
2025-02-11 13:50:04.777
at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488)
2025-02-11 13:50:04.777
at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:465)
2025-02-11 13:50:04.777
at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
2025-02-11 13:50:04.777
at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
2025-02-11 13:50:04.777
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2025-02-11 13:50:04.777
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

2025-02-11 13:50:04.777
at java.base/java.lang.Thread.run(Thread.java:1583)
```