humanmade / humanmade/Cavalcade-Runner

Jobs show "failed" when service is restarted.

Open
#54 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
PHP
Stars
56
Forks
27
PR merge metrics
No merged PRs in 30d

Description

When the cavalcade service is restarted, we do two things:

- Ignore the signal in any `wp cavalcade run` processes
- Wait for all running workers to complete in the cavalcade-runner

This works well to let the jobs complete, but the status of the process is changed, and cavalcade-runner interprets it as a fail.

[`is_done`](https://github.com/humanmade/Cavalcade-Runner/blob/master/inc/class-runner.php#L373) will return `true` here, but `shutdown()` will return `-1`. This is because (it seems) once a process has been sent `SIGTERM`, `proc_get_status` will return:

```
(
[command] => wp cavalcade run 440 --url='example.com/'
[pid] => 13589
[running] =>
[signaled] => 1
[stopped] =>
[exitcode] => -1
[termsig] => 15
[stopsig] => 0
)
```

(see `exitcode`)

According to the PHP docs: "The exit code returned by the process (which is only meaningful if running is FALSE). Only first call of this function return real value, next calls return -1." I think this might be an undocumented side-effect of a process ending with `SIGTERM`.

I think we need to have some logic to handle the case when `signaled => 1` or `stopsig => 15`, and maybe return `0` instead of `-1` in those cases?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in inc/class-runner.php, especially is_done() and shutdown(), and review how proc_get_status() values are interpreted after a worker receives SIGTERM. Reproduce a cavalcade service restart while a job is running and verify that the completed job is not reported as failed when the process is signaled.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.