Improve State restart time on failure caused by server restart, by using activity heartbeat, when startToClose is large
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 661
- Forks
- 63
- PR merge metrics
- No merged PRs in 30d
Description
Currently when iwf server restarts, the state api will fail and wait for next attempt by the startToClose timeout + backoff retry interval.
If the startToClose timeout is very large (e.g. >10 mins), it will wait for a long time. To avoid the unnecessary waiting, Temporal/Cadence has a concept of "activity heartbeat" to tell Temporal/Cadence server that the worker is still alive. If no heartbeat is received within heartbeat timeout, Temporal/Cadence will reschedule next activity immediately based on backoff retry policy.
Note: this is also because of the fact that Temporal/Cadence activity task/worker is "polling based". iWF task/worker is "pushing" so it doesn't have such issues.
Need to add a side thread(gorotine) in the activity code:
go (){
sleep(10 mins)
activity.heartbeat()
}
^^ is simplified code. We also need to cancel the goroutine when the activity is finished (so need to use golang channel and timer), to avoid goroutine leaks.
Maybe make 10mins configurable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the activity code and trace how state API failures, startToClose, and backoff retries are handled. Review the activity lifecycle and use the issue's heartbeat and cancellation requirements as the acceptance criteria, including avoiding goroutine leaks and considering whether the heartbeat interval should be configurable. No specific files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100