Request to Purge Scheduled Tasks

Hello,

We ran into an odd issue with our st2 environment (2.5.1). On Friday we noticed that chatops commands weren't publishing results. We decided it wasn't too important and we'd worry about it after the holiday weekend.

Today I poked around and couldn't find any errors in the logs and reloading didn't help, either. So I tried doing:

$ st2ctl stop
$ st2ctl start

And that did the trick - a little too well, though. st2 became flooded with every scheduled task for the past 4 days. Running st2 execution list had the EIDs scrolling too fast to keep track of (and the command limits retrieval to 100), so I quickly lost track of all scheduled tasks. I wasn't going to have st2 hammer away at all of these tasks, either.

I remembered that st2 can purge items from mongo, so I modified the executions.py file to purge everything not in DONE_STATES. This did the trick and we're now back up and running.

I really have no idea what caused this problem, so I'm not sure if this is a rare occurrence or if it happens often enough (where "often" is "it could possibly happen to anyone") that adding support to purge queued/scheduled items would be worthwhile.

Start by reading st2common/st2common/garbage_collection/executions.py, especially its handling of DONE_STATES, and review the st2ctl restart and st2 execution list behavior described in the report. Clarify how queued and scheduled executions should be selected for purging and define the expected behavior for the backlog scenario; the work is complete when that behavior is implemented and verifiable.