temporalio / temporalio/temporal
Support log-less graceful shutdown without "Error looking up host for shardID" errors
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
Today shutting down a single-binary server gives something like:
2023-05-11T14:45:49.683-0700 ERROR Error looking up host for shardID {"component": "shard-controller", "address": "127.0.0.1:33401", "error": "Not enough hosts to serve the request", "operation-result": "OperationFailed", "shard-id": 1, "logging-call-at": "controller_impl.go:387"}
go.temporal.io/server/common/log.(*zapLogger).Error
/home/runner/go/pkg/mod/go.temporal.io/server@v1.20.1/common/log/zap_logger.go:150
go.temporal.io/server/service/history/shard.(*ControllerImpl).acquireShards.func2
/home/runner/go/pkg/mod/go.temporal.io/server@v1.20.1/service/history/shard/controller_impl.go:387
go.temporal.io/server/service/history/shard.(*ControllerImpl).acquireShards.func3
/home/runner/go/pkg/mod/go.temporal.io/server@v1.20.1/service/history/shard/controller_impl.go:427
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-tq-scanner-taskqueue-0", "WorkerID": "431854@monolith@", "WorkerType": "WorkflowWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-processor-parent-close-policy", "WorkerID": "431854@monolith@", "WorkerType": "WorkflowWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-history-scanner-taskqueue-0", "WorkerID": "431854@monolith@", "WorkerType": "ActivityWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-tq-scanner-taskqueue-0", "WorkerID": "431854@monolith@", "WorkerType": "ActivityWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-batcher-taskqueue", "WorkerID": "431854@monolith@", "WorkerType": "ActivityWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
2023-05-11T14:45:50.685-0700 WARN Failed to poll for task. {"service": "worker", "Namespace": "temporal-system", "TaskQueue": "temporal-sys-history-scanner-taskqueue-0", "WorkerID": "431854@monolith@", "WorkerType": "WorkflowWorker", "Error": "error reading from server: EOF", "logging-call-at": "internal_worker_base.go:308"}
Describe the solution you'd like
Any solution that does not make a user think there is an error when they see logs. If this needs to be only half-done here and then something done at https://github.com/temporalio/cli to properly shutdown or swallow or something, no prob.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with service/history/shard/controller_impl.go around the logged calls at lines 387 and 427, then trace the single-binary shutdown path and related worker poll warnings. Check whether temporalio/cli is involved; done means graceful shutdown no longer emits error-like logs for expected teardown failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100