Physical Files Validation in a Big Dataset
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
We have a problem with a native API for physical files validation of a big dataset. $SERVER_URL/api/admin/validate/dataset/files/{datasetId} The API fails randomly on many files with an error "Failed to open datafile id {file_id} for reading". For example, we have a dataset with around 15000 files. The API fails on around 9000 datasets with failed to open error. Then we run validation file by file sequentially for this dataset using curl -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/admin/validateDataFileHashValue/{fileId}" all files are validated properly. We are using Dataverse 6.2 version.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing POST /api/admin/validate/dataset/files/{datasetId} against a dataset of roughly 15,000 files, then compare it with the per-file /api/admin/validateDataFileHashValue/{fileId} endpoint. Trace the bulk validation path and its "Failed to open datafile" failures; done means bulk validation completes successfully for the files that validate individually.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100