Provide ability to track a bulk imported file
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
This is similar to #650 but different enough I thought it warranted a separate ticket. The is related to the 1.x versions.
Basically the problem is being able to absolutely verify that a bulk imported file was successfully loaded into the system. This requires being able to determine what the file is renamed to during the bulk import process. Given that information we would be able to scan the accumulo.metadata table to find its matching entry. We realize that there is a race condition here in which the GC could have removed it before verification could take place. That situation could be handled by looking in the GC logs which is not very clean but doable. We could of course monitor the master log to determine the file mapping as well but I was hoping for a cleaner solution.
One possibility is to actually include the name of the original file in the key or value within the file column family of the accumulo metadata. Another possibility is to have the master pass back the list of file name mappings to the client. The later could be achieved by writing a mapping file into the directory that was being imported or alternatively the failure directory.
Contributor guide
Research direction
Start by tracing the bulk import process and how it records entries in the accumulo.metadata table. Compare the proposed original-file mapping, client-returned mappings, and mapping-file approaches, including the stated GC and master-log race concerns. Done means selecting and implementing a defined way to verify the imported file mapping, with validation of the failure and GC cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100