CC Workers can get blocked when jobs in the queue block for a long time

Issue

CC Workers can get blocked when jobs in the queue block for a long time.

Context

We've seen CATs failures, because staging took longer than 240s. It took longer than 240s, because CC Workers could not complete the staging jobs fast enough. They could not complete staging jobs fast enough, because they were busy completing other jobs. They were specifically busy with DeleteJobs deleting droplets in the blobstore. Deleting droplets from the blobstore is normally very fast. However, in GCP (RelInts hagrid env) we saw GCP response times of 360s! That means that those CC Worker jobs were blocked for 6 minutes. Even with multiple CC Workers, this can eventually block the whole job queue and other jobs are delayed to a point where staging is heavily impacted.

We saw the same issues of clogging the whole job queue in IBM some months ago, when the IBM Cloud Object Storage had degraded performance.

We noticed this in particular when using the Bits-Service as a blobstore.

Steps to Reproduce

It's quite difficult to reproduce that in fact. RelInt team has an environment (hagrid) which can provoke the situation when running CATs multiple times. Usually, the queue gets clogged on every other CAT run.

Expected result

Staging takes about a minute.

Current result

Staging takes more than 4 minutes.

Possible Fix

While we can introduce timeouts both in the Bits-Service and Bits-Service client gem, the CC Workers are the right layer to make the decision when to cancel a job and re-schedule it. More concretely, the BlobstoreDelete job should use a timeout to defend itself against slow Deletes. Deletes should usually not take more than 10s (conservatively), in 99% less than 1s.

Start by locating the CC Worker implementation of the BlobstoreDelete job and reviewing how it handles slow blobstore deletes. Check the Bits-Service and Bits-Service client gem context described in the issue, then determine how a delete timeout should cancel and reschedule the job. Done means slow deletes no longer clog the queue and staging remains near the expected duration.