WorkerCluster occasionally crashing.

Hey guys,

Very rarely (maybe once a month) I'll get this on one of our containers and it crashes that server. I have to restart the server to get it to respond. I'm not sure if it's a socketcluster problem but non of our other services experience this. We have 4 application servers in our SCC cluster. It doesn't affect users really because the rest of the cluster takes care of the load. Thanks!

1503450314685 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314685 - Worker 0 exited - Exit code: 1
1503450314685 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 6 exited - Exit code: 1
1503450314957 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 5 exited - Exit code: 1
1503450314957 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 3 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 2 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 1 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 7 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Start by tracing the WorkerCluster worker-spawn path and the container environment around the reported spawn /usr/bin/nodejs ENOENT error. Reproduce or isolate the conditions that make workers exit and the server stop responding, then verify that WorkerCluster remains responsive when workers fail.