IQSS / IQSS/dataverse

Checksum (MD5) calculation for local upload is slow

Open
#9,166 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: File Upload & Handling Type: Bug User Role: Depositor
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

What steps does it take to reproduce the issue?
Using the DVUploader, upload a large file (> 10GB) to Dataverse (local file-system, not S3).

  • When does this issue occur?
    Every time I upload a file to local storage.

  • Which page(s) does it occurs on?
    I'm exclusively using DVUploader.

  • What happens?
    On our DV host, the file first lands in /tmp. It is then copied to /usr/local/dv-temp/temp at which point it is unzipped to this same directory (we double-zip). At this point there are (3) copies of the file on local storage. Next I see that iostat shows a long running ~30MB/s read operation. I believe this corresponds with DV calculating an MD5 checksum. Running the md5sum Linux utility on the same file proceeds at 500MB/s -- so there is decent available performance on our DV host. Looking at the code (https://github.com/IQSS/dataverse/blob/1435dcca970ee524ec32506f1d8d50c81026fe86/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java), it appears that the checksum is calculated 1KB at a time, without using buffered IO -- which could explain the suboptimal performance that I think I am seeing.

  • To whom does it occur (all users, curators, superusers)?
    Anyone performing an upload.

  • What did you expect to happen?
    I would expect the md5 checksum calculation performance to be similar to the performance achieved by the md5sum linux utility.

Which version of Dataverse are you using?
5.10.1

Any related open or closed issues to this bug report?
I don't believe so.

Screenshots:

No matter the issue, screenshots are always welcome.

To add a screenshot, please use one of the following formats and/or methods described here:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java and trace the checksum path used by DVUploader for local filesystem uploads. Compare its large-file read performance with the Linux md5sum utility; done means checksum calculation is substantially faster without changing upload results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.