timeout errors when orchestrating syncs from prefect

### Helm Chart Version

1.8.3

### What step the error happened?

During the Sync

### Relevant information

We have airbyte deployed on airbyte EKS and the syncs are triggered by prefect using [this](https://github.com/Ishankoradia/prefect-airbyte) . For some random pipelines we keep running into timeout errors because airbyte didn't respond with a job_id.

Temporal showed these events around the same time

```2025-12-20T20:21:45.634192072Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.634Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.685215836Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.685Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.750847724Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.750Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.791274698Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.791Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.901732783Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.901Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.964411355Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.964Z","msg":"history client encountered error","service":"frontend","error":"consistent query buffer is full, this may be ca
used by too many queries and workflow not able to process query fast enough","service-error-type":"serviceerror.ResourceExhausted","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/c
lient/history/metric_client.go:103"}
2025-12-20T20:21:45.979344465Z stdout F {"level":"info","ts":"2025-12-20T20:21:45.979Z","msg":"Dropped long duration scheduled task.","shard-id":2,"address":"0.0.0.0:7234","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"connection_manager_f31eab8c-c993-4473-84e6-f9f61aecfc2b","wf-run-id":"77783bb6-370d-45a3-9cb7-d18b89777577","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"connection_manager_f31eab8c-c993-4473-84e6-f9f61aecfc2b","wf-run-id":"77783bb6-370d-45a3-9cb7-d18b89777577","queue-task-key":{"FireTime":"2125-11-26T20:21:45.97923554Z","TaskID":0},"queue-task-type":"UserTimer","queue-task":"9acea73b-9ef8-471b-9ca5-77f790fe6082/connection_manager_f31eab8c-c993-4473-84e6-f9f61aecfc2b/77783bb6-370d-45a3-9cb7-d18b89777577","wf-history-event-id":61,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/log/lazy_logger.go:58"}
```

Here is the prefect log stack

```Beginning subflow run 'lilac-prawn' for flow 'run-connection-sync'
01:32:37 AM
prefect.flow_runs
Triggering Airbyte Connection f31eab8c-c993-4473-84e6-f9f61aecfc2b, in workspace at '[http:///api/v1'](http:///api/v1')
01:32:42 AM
trigger-473
prefect.task_runs
Starting tenacity lib attempt 1
01:32:44 AM
trigger-473
prefect.task_runs
01:34:44 AM
trigger-473
prefect.task_runs
Attempt 1 failed with exception: Something went wrong while triggering sync for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b: . Retrying...
01:34:44 AM
trigger-473
prefect.task_runs
Starting tenacity lib attempt 2
01:34:54 AM
trigger-473
prefect.task_runs
01:36:54 AM
trigger-473
prefect.task_runs
Attempt 2 failed with exception: Something went wrong while triggering sync for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b: . Retrying...
01:36:54 AM
trigger-473
prefect.task_runs
Starting tenacity lib attempt 3
01:37:04 AM
trigger-473
prefect.task_runs
01:39:05 AM
trigger-473
prefect.task_runs
Attempt 3 failed with exception: Something went wrong while triggering sync for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b: . Retrying...
01:39:05 AM
trigger-473
prefect.task_runs
Task run failed with exception: RetryError() - Retries are exhausted
Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
return await self._connection.handle_async_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
raise exc
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
) = await self._receive_response_headers(**kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
data = await self._network_stream.read(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 32, in read
with map_exceptions(exc_map):
File "/home/ddp/.local/share/uv/python/cpython-3.10.12-linux-aarch64-gnu/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 192, in trigger_manual_sync_connection
response = await self._client.post(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1859, in post
return await self.request(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
File "/home/ddp/.local/share/uv/python/cpython-3.10.12-linux-aarch64-gnu/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
result = await fn(*args, **kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 225, in trigger_manual_sync_connection
raise Exception(
Exception: Something went wrong while triggering sync for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1337, in run_context
yield self
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1414, in run_task_async
await engine.call_task_fn(txn)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1360, in call_task_fn
result = await call_with_parameters(self.task.fn, parameters)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/connections.py", line 389, in trigger
) = await airbyte_client.trigger_manual_sync_connection(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
return await fn(*args, **kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
do = self.iter(retry_state=retry_state)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[]
01:39:05 AM
trigger-473
prefect.task_runs
Finished in state Failed('Task run encountered an exception RetryError: RetryError[]')
01:39:05 AM
trigger-473
prefect.task_runs
Encountered exception during execution: RetryError()
Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
return await self._connection.handle_async_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
raise exc
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
) = await self._receive_response_headers(**kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
data = await self._network_stream.read(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 32, in read
with map_exceptions(exc_map):
File "/home/ddp/.local/share/uv/python/cpython-3.10.12-linux-aarch64-gnu/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 192, in trigger_manual_sync_connection
response = await self._client.post(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1859, in post
return await self.request(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
File "/home/ddp/.local/share/uv/python/cpython-3.10.12-linux-aarch64-gnu/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
result = await fn(*args, **kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 225, in trigger_manual_sync_connection
raise Exception(
Exception: Something went wrong while triggering sync for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/flow_engine.py", line 1325, in run_context
yield self
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/flow_engine.py", line 1387, in run_flow_async
await engine.call_flow_fn()
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/flow_engine.py", line 1339, in call_flow_fn
result = await call_with_parameters(self.flow.fn, self.parameters)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/flows.py", line 62, in run_connection_sync
airbyte_sync = await task(airbyte_connection.trigger.aio)(airbyte_connection)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1416, in run_task_async
return engine.state if return_type == "state" else await engine.result()
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1016, in result
raise self._raised
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1337, in run_context
yield self
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1414, in run_task_async
await engine.call_task_fn(txn)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect/task_engine.py", line 1360, in call_task_fn
result = await call_with_parameters(self.task.fn, parameters)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/prefect_airbyte/connections.py", line 389, in trigger
) = await airbyte_client.trigger_manual_sync_connection(
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
return await fn(*args, **kwargs)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
do = self.iter(retry_state=retry_state)
File "/mnt/appdata/prefect-proxy/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[]
```

The sync on airbyte side fails with a platform error.

We are using this [api](https://github.com/Ishankoradia/prefect-airbyte/blob/5cdbf91716b53b273f8892863e9d05f82a37f915/prefect_airbyte/client.py#L175) to trigger airbyte sync and this [api](https://github.com/Ishankoradia/prefect-airbyte/blob/5cdbf91716b53b273f8892863e9d05f82a37f915/prefect_airbyte/client.py#L418) to poll for status of the job.

Have all resources and graphs on our EKS deployment but everything is under control. We have been facing this for quite sometime and haven't been able to figure out the way around this.

Any help from the airbyte team or the community is highly appreciated

### Relevant log output

```shell
Warning from airbyte_platform: Something went wrong within the airbyte platform

Activity with activityType='CreateNewAttemptNumber' failed: 'Activity task failed'. scheduledEventId=85, startedEventId=94, activityId=c69ab741-ea31-30bb-bfa0-2d059d5ee905, identity='1@airbyte-worker-7dc57677cf-dlxsv', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

2025-12-21 01:51:44 platform INFO Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=1, totalPartialFailureLimit=1, successiveCompleteFailures=1, successivePartialFailures=0, totalCompleteFailures=1, totalPartialFailures=0)
Backoff before next attempt: 10 seconds
2025-12-21 01:51:44 platform ERROR Failing job: 62971, reason: Job failed after too many retries for connection f31eab8c-c993-4473-84e6-f9f61aecfc2b
```