temporalio / temporalio/temporal
Aim for error free rolling bounces and upgrades of Temporal
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Currently a rolling bounce or rolling upgrade of Temporal results in lots of error messages related to shard stealing. Error messages include:
Failed to update shard
Error updating ack level for shard
Error updating timer ack level for shard
These errors make it harder to identify real issues during a deployment and lead operators to tend to ignore these potentially important error messages.
Ideally, we should be able to do a rolling bounce or rolling upgrade of Temporal without encountering these errors on a routine basis.
One thing to keep in mind here is that Temporal is very often deployed using Kubernetes. Any solution should be compatible with the default Kubernetes Deployment RollingUpdate strategy.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the shard-stealing paths associated with the reported "Failed to update shard" and ack-level errors during a rolling bounce or upgrade. Compare their behavior with Kubernetes Deployment RollingUpdate, and consider the work complete when routine rolling changes no longer produce these errors or obscure genuine deployment problems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100