Ability to publish to S3 buckets with Object Lock by including md5 checksum.

New feature

AWS S3 Object Lock is a feature in Amazon Simple Storage Service (S3) that allows you to protect objects from being deleted or overwritten for a specified retention period. Some orgs will use Object Lock to meet data retention or regulatory requirements.

The PutObject call to a bucket with Object Lock requires an extra Content-MD5 header (docs). Currently, publishing an object to a bucket with Object Lock returns an error:

Error executing process > 'MYPROCESS (1)'

Caused by:
  Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID: XXXX; S3 Extended Request ID: XXXX; Proxy: null)

Usage scenario

I'd like to publish a file to a bucket that has Object Lock turned on.

Suggest implementation

For large multipart uploads (often the case with Nextflow published files), the ETag does not represent the MD5 digest, so the object will likely need to be downloaded and hashed by Nextflow so that the Content-MD5 header can be attached to the publication event.

Start by tracing Nextflow’s S3 publication path, including the handling of large multipart uploads, and compare the current PutObject request with AWS Object Lock requirements. The work is done when publishing to an Object Lock-enabled bucket supplies a valid Content-MD5 or supported checksum and succeeds for the relevant upload paths.