Code offer: CloudWatch monitoring of job queue size

If anyone is interested, we've recently developed a small shell script designed to monitor the job queue and report metrics to Amazon CloudWatch.

#!/bin/sh

LOG_FILE=/var/log/ckan/ckan-worker-queue.log
TEMP_FILE=/tmp/ckan-worker-queue.log
echo "CKAN job queue at $(date):" >> $LOG_FILE
JOB_COUNT=$(/usr/lib/ckan/default/bin/paster --plugin=ckan jobs list -c /etc/ckan/default/production.ini 2>/dev/null | tee -a $LOG_FILE | tee $TEMP_FILE | wc -l)
echo "Total: $JOB_COUNT job(s)" >> $LOG_FILE
if [ "$JOB_COUNT" -gt 0 ]; then
OLDEST_TIME=$(date -u --date $(head -1 $TEMP_FILE | awk '{print $1}') +'%s')
MAX_AGE=$(expr $(date -u +'%s') - $OLDEST_TIME)
echo "Oldest job is from $OLDEST_TIME ($(expr $MAX_AGE / 60) minute(s) ago)" >> $LOG_FILE
else
MAX_AGE=0
fi
aws cloudwatch put-metric-data --region us-east-1 --namespace CKAN --metric-data "MetricName=job_queue_size,Dimensions=[{Name=Application,Value=MyCKANApplication},{Name=Environment,Value=DEV}],Value=$JOB_COUNT" "MetricName=max_job_age,Dimensions=[{Name=Application,Value=MyCKANApplication},{Name=Environment,Value=DEV}],Value=$MAX_AGE"

Note that `LOG_FILE` must be in a directory that exists and is writable by the user running the script. Also, the machine where this runs needs to have appropriate AWS credentials set up to allow access to CloudWatch.

The metric dimensions are optional; we use them because we have multiple different CKAN portals, and multiple environments (DEV, STAGING, etc), in the same AWS account.

Start with the supplied shell script, including the CKAN `paster --plugin=ckan jobs list` command and the AWS `cloudwatch put-metric-data` entry point. Check how this repository handles operational scripts or monitoring integrations; completion would require a defined project location, validated queue metrics, and documented writable log and AWS credential requirements.

Written by the indexing model from the issue text.