How to correctly handle locked and reprocessed batches?

Hi,

We are using the loader function to load JSON files (generated from an Apache Spark application running in an EMR cluster) to Redshift.

We ran into an issue where many batches became "locked". We think this was related to a problem where we had many ProvisionedThroughputExceededException on the DynamoDB tables backing the loader function. We increased the provisioning for the tables, but we are noticing that for some of our s3Prefix configurations, we are not seeing any new data loaded to Redshift. We are also seeing many of these logs in Cloudwatch:

"2017-03-16T20:00:20.166Z 2e92425c-0a83-11e7-8d55-69fee54689ef Reload of Configuration Complete after attempting to write to Locked Batch a0d645fd-5dbb-4386-b2c5-4674426851a9. Attempt 1"

(This log was for batchId a0d645fd-5dbb-4386-b2c5-4674426851a9 in particular, but we see them for many batches.)

I noticed that the batches for which we are seeing these messages were marked as "locked" in the LambdaRedshiftBatches table. I used the included unlockBatch.js script, and the reprocessBatch.js script, to attempt to "reprocess" these batches. This did not work - the batch status in LambdaRedshiftBatches became marked as "reprocessed", but we continued to see the aforementioned logs in Cloudwatch.

Next, I noticed that for the rows in LambdaRedshiftBatchLoadConfig where the s3Prefix corresponded to a locked/reprocessed batchId, the currentBatch property referred to the locked/reprocessed batchId. I queried LambdaRedshiftBatches for any open batches for that s3Prefix, then manually updated the currentBatch property in LambdaRedshiftBatchLoadConfig with the batchId of the open batch.

This seems to have resolved the issue, but I'm unsure now if I've missed loading the files in the locked/reprocessed batches into Redshift. I'm left with a number of other questions:

1. What does it mean for a batch to have the status "locked" or "reprocessed"?
2. How can batches get into the "locked" state?
3. I noticed that there were many "open" batches for a particular s3Prefix in LambdaRedshiftBatches. Is this normal? The documentation states "There will always be one open batch, and may be multiple closed batches per S3 input prefix from LambdaRedshiftBatchLoadConfig", but does this mean **only** one open batch per s3Prefix?
4. If it is true that there should be **only** one open batch per s3Prefix, how could the system get into a state where there are multiple?
4. Was there a better way for me to handle my situation?

Hope you can help, and please let me know if any more info from me would be of use.

Regards,
Justin