IQSS / IQSS/dataverse

Feature Request: Storing original file checksum

Open
#11,315 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Feature
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Overview of the Feature Request

For file management using external tools and for Dataverse SPA I would like to have the checksum of the original file (ingested) in the output of the API api/datasets/{id}/versions/{versionId}/files.

Example: /api/datasets/:persistentId/versions/:latest/files?persistentId=doi:10.11111/5QHIDC

{
	"description": "",
	"label": "TableS7_Fouchecourt.tab",
	"restricted": false,
	"directoryLabel": "test_DVU",
	"version": 3,
	"datasetVersionId": 54476,
	"dataFile": {
	"id": 51859,
	"persistentId": "",
	"filename": "TableS7_Fouchecourt.tab",
	"contentType": "text/tab-separated-values",
	"filesize": 181528,
	"description": "",
	"storageIdentifier": "inrae://dev-test:19547b38922-f6d1779da1f5",
	"originalFileFormat": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
	"originalFormatLabel": "MS Excel Spreadsheet",
	"originalFileSize": 267297,
	"originalFileName": "TableS7_Fouchecourt.xlsx",
	"UNF": "UNF:6:THbSVfLiqCuW/Dt51uiCIw==",
	"rootDataFileId": -1,
	"md5": "9e6bb0a0ada77e2e62221b148957532e",
	"checksum": {
		"type": "MD5",
		"value": "9e6bb0a0ada77e2e62221b148957532e"
	},
	"creationDate": "2025-02-27"
}

Should now contain :

"originalFileChecksum": {
	"type": "MD5",
	"value": "f2cdd3f069e59ea9749a641d822355d7"
},

Or be refactored as :

"originalFile": {
	"format": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
	"formatLabel": "MS Excel Spreadsheet",
	"size": 267297,
	"name": "TableS7_Fouchecourt.xlsx",
	"md5": "f2cdd3f069e59ea9749a641d822355d7",
	"checksum": {
		"type": "MD5",
		"value": "f2cdd3f069e59ea9749a641d822355d7"
	},
},

Specifications

In order to do this I see :

  • Adding related columns in database and java Object related code
  • Calculate the checksum on file upload and store it in database
  • Having the capacity to re-calculate all original files for existing repositories by having a new API /admin endpoint (must handle batch/limit for large number of file)
  • Modify the output of the API endpoint GET api/datasets/{id}/versions/{versionId}/files
  • Update of guides and having a release note

What inspired the request?

The handling of duplicated file for ingested file is done by comparing the original file name in Dataverse and DVUploader, this is not enough.
Related to https://github.com/GlobalDataverseCommunityConsortium/dataverse-uploader/issues/29

Are you thinking about creating a pull request for this feature?

Not in team priority

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the GET api/datasets/{id}/versions/{versionId}/files endpoint and the file-upload and database paths described in the specifications. Review how original file metadata is represented, then determine the scope for checksum calculation, storage, an admin batch endpoint, API output, guides, and release notes. Done means the agreed design is implemented across these areas and existing files are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, database, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.