humanmade / humanmade/Cavalcade-Runner
Jobs show "failed" when service is restarted.
- Dominant language
- PHP
- Stars
- 56
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
When the cavalcade service is restarted, we do two things:
- Ignore the signal in any `wp cavalcade run` processes
- Wait for all running workers to complete in the cavalcade-runner
This works well to let the jobs complete, but the status of the process is changed, and cavalcade-runner interprets it as a fail.
[`is_done`](https://github.com/humanmade/Cavalcade-Runner/blob/master/inc/class-runner.php#L373) will return `true` here, but `shutdown()` will return `-1`. This is because (it seems) once a process has been sent `SIGTERM`, `proc_get_status` will return:
```
(
[command] => wp cavalcade run 440 --url='example.com/'
[pid] => 13589
[running] =>
[signaled] => 1
[stopped] =>
[exitcode] => -1
[termsig] => 15
[stopsig] => 0
)
```
(see `exitcode`)
According to the PHP docs: "The exit code returned by the process (which is only meaningful if running is FALSE). Only first call of this function return real value, next calls return -1." I think this might be an undocumented side-effect of a process ending with `SIGTERM`.
I think we need to have some logic to handle the case when `signaled => 1` or `stopsig => 15`, and maybe return `0` instead of `-1` in those cases?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in inc/class-runner.php, especially is_done() and shutdown(), and review how proc_get_status() values are interpreted after a worker receives SIGTERM. Reproduce a cavalcade service restart while a job is running and verify that the completed job is not reported as failed when the process is signaled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100