Issues with regular dataset content replacement failures

Following on from ongoing support ticket discussions with Danny and Gustavo

Submitting new files is sometimes failing for us. We are seeing this in both Add and Replace. This is an inconsistent error (as can be seen in the version history on the datasets).

We are attempting to replace all the files in a data set as and when new data is available, sequentially processing each file and then publishing. This checks if the file is present in the dataverse and then replaces / uploads accordingly. Some files will fail on some occasions. Replace will also sometimes fail but remove the original. These can be seen in the mix of New / Replaced / Removed files on the version change log:

I think the "removed" thing may be a separate bug - with both replace and add failing on occasion (although the replace fails and ends up recording as a remove of the original file).

Example log snippet from application server:

2021-04-07 01:09:34 TRACE {"status":"OK","data":{"files":[{"description":"","label":"reported_cases.csv","restricted":false,"version":1,"datasetVersionId":238355,"dataFile":{"id":4499369,"persistentId":"","pidURL":"","filename":"reported_cases.csv","contentType":"text/csv","filesize":44048,"description":"","storageIdentifier":"s3://dvn-cloud:178a9e0bbd6-fd605ceb3702","rootDataFileId":4497776,"previousDataFileId":4497776,"md5":"c0a3bcccbed641fafda1567113cb1c20","checksum":{"type":"MD5","value":"c0a3bcccbed641fafda1567113cb1c20"},"creationDate":"2021-04-06"}}]}}
2021-04-07 01:10:03 TRACE file subnational/canada/cases/summary/rt_plot.png
2021-04-07 01:10:03 TRACE check if the file exists
2021-04-07 01:10:03 TRACE uploading file subnational/canada/cases/summary/rt_plot.png
2021-04-07 01:10:20 ERROR Error in dataverse::add_dataset_file(file = file_full_path, dataset = dataset_id): Bad Request (HTTP 400).

Interestingly this can be seen in a brand new dataset too (although why it duplicated an existing one I don't know): https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/ER7GJH

Start with the application-server log and reproduce sequential Add and Replace uploads against a dataset, comparing the version history with the reported HTTP 400 failure. Investigate both the failed upload and the case where Replace records the original as Removed; done means uploads complete consistently and the version history records the correct New or Replaced state.