action stuck in running after hard reset

SUMMARY

We encountered an issue where an action was stuck in the running state after a hard reset.

The hard reset exposed a race condition between the action_execution_d_b and live_action_d_b for a particular running action's status. After the system came up, the action_execution_d_b status was set to running and live_action_d_b was status set to succeeded. The action never gets cleaned up.

ISSUE TYPE

Pick one below and delete the rest:

• Bug Report

STACKSTORM VERSION

Paste the output of st2 --version: st2 2.9.1, on Python 2.7.15

OS / ENVIRONMENT / INSTALL METHOD

custom install

reproduced in docker/centos 7.5.1, custom install

STEPS TO REPRODUCE

Since this is a timing issue, it's almost impossible to catch in production (we only saw this once). I reproduced with the following steps:

1. run an action to completion
2. modify mongo db entry in action_execution_db status to running
3. verify action is running
4. restart st2
   • action is still running

EXPECTED RESULTS

After a restart, st2 should start with a clean slate. There's no possibility for an action to be running at startup.

ACTUAL RESULTS

Action was running after restart.

example:

[root@8137ae0c21ad /]# st2 execution get 5c403cf9ab3bb0197b7f7bfe
id: 5c403cf9ab3bb0197b7f7bfe
status: running (11631s elapsed) <-------------- running for 3+ hours
parameters:
[...]
[root@8137ae0c21ad /]# uptime
 19:45:43 up 4 min,  0 users,  load average: 0.62, 0.67, 0.30 <---- docker has been up for 4m

on the test system:

> db.action_execution_d_b.find({"_id":  ObjectId("5c403cf9ab3bb0197b7f7bfe")}, {"status":1,"_id":0}).pretty()
{ "status" : "running" }
> db.live_action_d_b.find({"_id":  ObjectId("5c403cf9ab3bb0197b7f7bfd"), {"status":1,"_id":0}}.pretty()
{ "status" : "succeeded" }

Start by reproducing the restart sequence with st2 execution get and inspect the action_execution_db and live_action_d_b records shown in the report. Trace startup handling for actions whose records disagree, and consider the work complete when a restart no longer leaves an action reported as running after its live record has succeeded.