Uningest of corrupted tab file
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
For preservation, we run checksum file integrity tests. We have some corrupted tab files. We would like to replace them from our backup without creating a new version of dataset. To do that we will need to uningest tabular file (delete all created metadata during ingest). But since tab file is corrupted the regular uningest will fail.
It would be good to have an API that can uningest corrupted tab files, i.e delete related metadata and return to original file. In this case the original file can be replace on filesystem and reingested again.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing uningest API and tabular-file ingest path, focusing on how ingest-created metadata is removed. Determine how the operation can proceed when the tab file is corrupted without creating a new dataset version. Done means corrupted tab files can be uningested, the related metadata is removed, and the original file can be replaced and reingested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100