Continued submission of tasks to an orphaned task queue can lead to Cassandra tombstone scanning issues

It is possible for poorly managed user executions to result in scenarios that cause Cassandra tombstone scanning issues.

We ran into one such scenario:

• an integration test ran every hour of every day. It created a cron workflow that fired every minute
• under normal circumstances, the test would verify that the cron workflow ran 2 or 3 times and then terminate the workflow
• but if the test crashed or there was test infrastructure failure, the workflow was never terminated, resulting in a cron that created a new workflow task every minute, forever
• these workflows were being submitted to unique task queues (every run of the test created a task queue with a uuid in the name)
• so there were never any workers to pick up these wf tasks and advance the cursor on the tasks table
• but since new tasks were being submitted by the orphaned cron every minute, the task manager for the task queue remained active and the taskReader would scan over all the tombstones resulting from the TTL'ed tasks on the queue

From speaking with Samar it sounds like one possible approach is to stop using TTL's to remove expired tasks from the tasks table and instead delete them explicitly. This would give the task reader an opportunity to move the cursor when it determined there were no active tasks beyond a certain point.