sync pulling 0 records & exits out without raising error
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Helm Chart Version
1.8.3
### What step the error happened?
During the Sync
### Relevant information
This is a standard connection of googel sheets (0.8.4) -> postgres (2.0.9). The source has some finite records and the connection is set full refresh overwrite.
This sync has been running fine for quite sometime. But faced this error recently
Temporal pod has been complaining
```{"level":"error","ts":"2025-12-18T00:03:06.056Z","msg":"Update workflow execution operation failed.","shard-id":3,"address":"0.0.0.0:7234","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62560","wf-run-id":"019b2ec3-4f80-7e6a-b993-6e4394c9b39d","store-operation":"update-wf-execution","error":"context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:188\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:654\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:502\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:852\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).executeUserTimerTimeoutTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:208\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:123\ngo.temporal.io/server/service/history/queues.(*activeStandbyExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/active_standby_executor.go:67\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/executable.go:374\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:62\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:63\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"warn","ts":"2025-12-18T00:03:06.056Z","msg":"Fail to process task","shard-id":3,"address":"0.0.0.0:7234","component":"timer-queue-processor","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62560","wf-run-id":"019b2ec3-4f80-7e6a-b993-6e4394c9b39d","queue-task-key":{"FireTime":"2025-12-18T00:02:50.731Z","TaskID":1086325909},"queue-task-type":"UserTimer","queue-task":"9acea73b-9ef8-471b-9ca5-77f790fe6082/replication_62560/019b2ec3-4f80-7e6a-b993-6e4394c9b39d","wf-history-event-id":17,"error":"context deadline exceeded","error-type":"persistence.AppendHistoryTimeoutError","unexpected-error-attempts":1,"lifecycle":"ProcessingFailed","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/log/lazy_logger.go:63"}
{"level":"error","ts":"2025-12-18T00:03:06.058Z","msg":"Update workflow execution operation failed.","shard-id":4,"address":"0.0.0.0:7234","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62561","wf-run-id":"019b2ec3-5643-7446-9a73-3532e0ec5615","store-operation":"update-wf-execution","error":"context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:188\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:654\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:502\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:852\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).executeUserTimerTimeoutTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:208\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:123\ngo.temporal.io/server/service/history/queues.(*activeStandbyExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/active_standby_executor.go:67\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/executable.go:374\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:62\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:63\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:211"}
{"level":"warn","ts":"2025-12-18T00:03:06.059Z","msg":"Fail to process task","shard-id":4,"address":"0.0.0.0:7234","component":"timer-queue-processor","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62561","wf-run-id":"019b2ec3-5643-7446-9a73-3532e0ec5615","queue-task-key":{"FireTime":"2025-12-18T00:02:52.433Z","TaskID":1219494862},"queue-task-type":"UserTimer","queue-task":"9acea73b-9ef8-471b-9ca5-77f790fe6082/replication_62561/019b2ec3-5643-7446-9a73-3532e0ec5615","wf-history-event-id":17,"error":"context deadline exceeded","error-type":"persistence.AppendHistoryTimeoutError","unexpected-error-attempts":1,"lifecycle":"ProcessingFailed","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/log/lazy_logger.go:63"}
{"level":"warn","ts":"2025-12-18T00:03:07.037Z","msg":"List query exceeded threshold","duration":1.383039247,"visibility-query":"ExecutionStatus != 'Running'","namepsace":"default","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/visibility/visiblity_manager_metrics.go:146"}
{"level":"info","ts":"2025-12-18T00:03:23.427Z","msg":"Stopped physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Stopped","cause":"Idle","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/matching/physical_task_queue_manager.go:212"}
{"level":"info","ts":"2025-12-18T00:03:32.833Z","msg":"Stopped physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Stopped","cause":"Idle","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/matching/physical_task_queue_manager.go:212"}
{"level":"info","ts":"2025-12-18T00:03:48.878Z","msg":"Stopped physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/CHECK_CONNECTION/1","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Stopped","cause":"Idle","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/matching/physical_task_queue_manager.go:212"}
{"level":"info","ts":"2025-12-18T00:04:18.428Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/matching/physical_task_queue_manager.go:194"}
```
Sync logs
```2025-12-18 05:31:33 platform INFO APPLY Stage: BUILD — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO APPLY Stage: CLAIM — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO Claimed: true for workload a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover via API in dataplane AUTOa90964a4-3e4b-4ea9-9764-679f65533ecf (d3db3e5a-91f6-405a-9891-b3cab03ddbb6)
2025-12-18 05:31:33 platform INFO APPLY Stage: LOAD_SHED — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO No pod found running for workload a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover
2025-12-18 05:31:33 platform INFO APPLY Stage: MUTEX — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO No mutex key specified for workload: a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover. Continuing...
2025-12-18 05:31:33 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO APPLY Stage: LAUNCH — (workloadId=a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover)
2025-12-18 05:31:33 platform INFO [initContainer] image: docker.io/airbyte/workload-init-container:1.8.3 resources: ResourceRequirements(claims=[], limits={}, requests={}, additionalProperties={})
2025-12-18 05:31:44 platform INFO Attempting to update workload: a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover to LAUNCHED.
2025-12-18 05:31:44 platform INFO Pipeline completed for workload: a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover.
2025-12-18 05:31:50 platform INFO
2025-12-18 05:31:50 platform INFO Connector exited, processing output
2025-12-18 05:31:51 platform INFO ----- START DISCOVER -----
2025-12-18 05:31:51 platform INFO Output file jobOutput.json found
2025-12-18 05:31:51 platform INFO Connector exited with exit code 0
2025-12-18 05:31:51 platform INFO Reading messages from protocol version 0.2.0
2025-12-18 05:31:51 platform INFO
2025-12-18 05:31:51 platform INFO Running discovery on sheet 13NbFN9uZoD6L60ARXGDCVSrQF0k-4F8amED-L0MlnXY
2025-12-18 05:31:51 platform INFO Checking for optional control message...
2025-12-18 05:31:51 platform INFO Writing catalog result to API...
2025-12-18 05:31:51 platform INFO Finished writing catalog result to API.
2025-12-18 05:31:51 platform INFO Writing output of a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover to the doc store
2025-12-18 05:31:51 platform INFO Marking workload a0d20682-1a0a-4d94-adb3-8db965b4f5d4_1766016000000_discover as successful
2025-12-18 05:31:51 platform INFO Deliberately exiting process with code 0.
2025-12-18 05:31:51 platform INFO
2025-12-18 05:31:51 platform INFO ----- END DISCOVER -----
2025-12-18 05:31:51 platform INFO
2025-12-18 05:31:52 platform INFO APPLY Stage: BUILD — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO APPLY Stage: CLAIM — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO Claimed: true for workload 8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync via API in dataplane AUTOa90964a4-3e4b-4ea9-9764-679f65533ecf (d3db3e5a-91f6-405a-9891-b3cab03ddbb6)
2025-12-18 05:31:52 platform INFO APPLY Stage: LOAD_SHED — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO No pod found running for workload 8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync
2025-12-18 05:31:52 platform INFO APPLY Stage: MUTEX — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO Mutex key: 8c10e782-f808-4428-bb84-89b4bf70b46f specified for workload: 8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync. Attempting to delete existing pods...
2025-12-18 05:31:52 platform INFO Mutex key: 8c10e782-f808-4428-bb84-89b4bf70b46f specified for workload: 8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync found no existing pods. Continuing...
2025-12-18 05:31:52 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO APPLY Stage: LAUNCH — (workloadId=8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync)
2025-12-18 05:31:52 platform INFO [initContainer] image: docker.io/airbyte/workload-init-container:1.8.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=2000m}, requests={memory=2Gi, cpu=1000m}, additionalProperties={})
2025-12-18 05:31:52 platform INFO Launching replication pod: replication-job-62561-attempt-0 (selectors = {type=jobs}) with containers:
2025-12-18 05:31:52 platform INFO [source] image: docker.io/airbyte/source-google-sheets:0.8.4 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=2000m}, requests={memory=2Gi, cpu=1000m}, additionalProperties={})
2025-12-18 05:31:52 platform INFO [destination] image: docker.io/airbyte/destination-postgres:2.0.9 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=2000m}, requests={memory=2Gi, cpu=1000m}, additionalProperties={})
2025-12-18 05:31:52 platform INFO [orchestrator] image: docker.io/airbyte/container-orchestrator:1.8.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=2000m}, requests={memory=2Gi, cpu=1000m}, additionalProperties={})
2025-12-18 06:02:51 platform INFO
----- START POST REPLICATION OPERATIONS -----
2025-12-18 06:02:51 platform INFO No post-replication operation(s) to perform.
2025-12-18 06:02:51 platform INFO
----- END POST REPLICATION OPERATIONS -----
2025-12-18 06:08:52 platform INFO Attempting to update workload: 8c10e782-f808-4428-bb84-89b4bf70b46f_62561_0_sync to LAUNCHED.
2025-12-18 06:08:52 platform WARN Failed to update workload status to launched. Workload may be reprocessed on restart.Exception: io.airbyte.api.client.ApiException: Gone
message: Gone
stackTrace: [Ljava.lang.StackTraceElement;@7ee85718
```
### Relevant log output
```shell
```
Contributor guide
Assessment
This issue has not been assessed yet.