Feature Request: Optionally send Content-MD5 header when uploading to S3-compatible storage

Overview of the Feature Request
When uploading files to S3-compatible storage, Dataverse should support sending the Content-MD5 header so the server can check the integrity of the upload. This is needed to support so-called immutable buckets with Versioning and ObjectLock enabled. Currently, all uploads to buckets with ObjectLock enabled fail with a 400 Bad Request along the lines of:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidRequest</Code>
  <Message>Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters</Message>
  <RequestId>{{ SNIP }}</RequestId>
  <HostId>{{ SNIP }}</HostId>
</Error>

We have tested all of the following combinations with Dataverse 6.7, and all fail the same way:

• Uploads via python-DVUploader and the Dataverse UI
• With upload-redirect both true and false.

To support this, the API endpoint that generates pre-signed URLs would need to accept the md5 checksum of the object to be uploaded. This would be included in the generated pre-signed URL (as a signed header) and submitted in the Content-MD5 header of whichever client ultimately submits the object to S3.

Some AWS-specific documentation is available here. Admittedly, like much of AWS's documentation, it is not especially clear and the details seem to be buried in the SDKs. We will note that the behavior does not seem specific to CloudianS3's implementation, though we haven't confirmed that ourselves. Our observation that Dataverse does not send the Content-MD5 header to S3 is true regardless of the backend implementation (AWS, Cloudian, minio, etc.).

What kind of user is the feature intended for?

• Systems administrators gain the ability to use immutable S3 buckets.
• Depositors and curators benefit from additional integrity checks on uploaded files.

What inspired the request?
We are integrating Berkeley's Dataverse 6.7 with a CloudianS3 backend as part of a pilot to provide very large scale research data support. Per our usual security practice, we initially attempted to use a bucket with ObjectLock enabled, but this failed with the error described above. This has also led us to wonder about what integrity checks could be implemented at upload time when using S3 backends.

What existing behavior do you want changed?

• For backends using direct uploads:
  • Presigned URLs should support receiving the checksum of a file to be uploaded, to be included in the Content-MD5 header and as part of the request signature.
  • The Web UI should support sending file checksums as part of the upload process for collections using an S3 backend.
  • Ancillary tooling (e.g. python-dvuploader) should also support submitting the checksum to the Dataverse API as well as to the S3 backend when direct-uploading the file.
• For backends NOT using direct uploads:
  • The server should (perhaps optionally) compute and submit the Content-MD5 header when uploading files from the Dataverse server to S3 storage.

Are you thinking about creating a pull request for this feature?

No, sorry—we don't have the resources to develop this feature. But we could assist with testing with our CloudianS3 ("HyperStore") backend.

What workarounds are available?

• To verify file integrity, it seems necessary to perform out-of-band audits comparing Dataverse's record of file checksums with those produced by a ComputeChecksum report (or similar) on the bucket. Organizations may wish to do this anyway as part of their bitrot mitigation strategy.
• To support bucket immutability, we have found it possible to replicate to a bucket with ObjectLock enabled. The trade-off is that this at least doubles storage usage, though that is probably priced in already as part of an organization's disaster recovery and backup strategy. Since ObjectLock (but not Versioning) can be toggled after a bucket is created, it would still be possible to fail over to the replicated bucket by temporarily disabling ObjectLock.

Edit: Perusing the source with @anarchivist this morning, we think the primary integration point would be in getMPUploadUrls. That would need to be adapted to accept the checksum up front, and the client would need to calculate and submit the checksum when submitting the request to generate the presigned URL. (Currently, the client calculates the checksum after the upload to S3 is completed as part of the submission to Dataverse.)

Start at getMPUploadUrls in src/main/java/edu/harvard/iq/dataverse/api/Datasets.java and trace how the Web UI and python-DVUploader request presigned URLs and calculate checksums. Review the direct-upload and server-upload paths described in the issue. Done means the checksum can be accepted before upload and used for Content-MD5 handling across the requested clients and storage paths.