Feature Request: Storing original file checksum
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Overview of the Feature Request
For file management using external tools and for Dataverse SPA I would like to have the checksum of the original file (ingested) in the output of the API api/datasets/{id}/versions/{versionId}/files.
Example: /api/datasets/:persistentId/versions/:latest/files?persistentId=doi:10.11111/5QHIDC
{
"description": "",
"label": "TableS7_Fouchecourt.tab",
"restricted": false,
"directoryLabel": "test_DVU",
"version": 3,
"datasetVersionId": 54476,
"dataFile": {
"id": 51859,
"persistentId": "",
"filename": "TableS7_Fouchecourt.tab",
"contentType": "text/tab-separated-values",
"filesize": 181528,
"description": "",
"storageIdentifier": "inrae://dev-test:19547b38922-f6d1779da1f5",
"originalFileFormat": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"originalFormatLabel": "MS Excel Spreadsheet",
"originalFileSize": 267297,
"originalFileName": "TableS7_Fouchecourt.xlsx",
"UNF": "UNF:6:THbSVfLiqCuW/Dt51uiCIw==",
"rootDataFileId": -1,
"md5": "9e6bb0a0ada77e2e62221b148957532e",
"checksum": {
"type": "MD5",
"value": "9e6bb0a0ada77e2e62221b148957532e"
},
"creationDate": "2025-02-27"
}
Should now contain :
"originalFileChecksum": {
"type": "MD5",
"value": "f2cdd3f069e59ea9749a641d822355d7"
},
Or be refactored as :
"originalFile": {
"format": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"formatLabel": "MS Excel Spreadsheet",
"size": 267297,
"name": "TableS7_Fouchecourt.xlsx",
"md5": "f2cdd3f069e59ea9749a641d822355d7",
"checksum": {
"type": "MD5",
"value": "f2cdd3f069e59ea9749a641d822355d7"
},
},
Specifications
In order to do this I see :
- Adding related columns in database and java Object related code
- Calculate the checksum on file upload and store it in database
- Having the capacity to re-calculate all original files for existing repositories by having a new API
/adminendpoint (must handle batch/limit for large number of file) - Modify the output of the API endpoint GET
api/datasets/{id}/versions/{versionId}/files - Update of guides and having a release note
What inspired the request?
The handling of duplicated file for ingested file is done by comparing the original file name in Dataverse and DVUploader, this is not enough.
Related to https://github.com/GlobalDataverseCommunityConsortium/dataverse-uploader/issues/29
Are you thinking about creating a pull request for this feature?
Not in team priority
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the GET api/datasets/{id}/versions/{versionId}/files endpoint and the file-upload and database paths described in the specifications. Review how original file metadata is represented, then determine the scope for checksum calculation, storage, an admin batch endpoint, API output, guides, and release notes. Done means the agreed design is implemented across these areas and existing files are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, database, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100