apache / apache/accumulo

Provide ability to track a bulk imported file

Open
#1,963 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

This is similar to #650 but different enough I thought it warranted a separate ticket. The is related to the 1.x versions.

Basically the problem is being able to absolutely verify that a bulk imported file was successfully loaded into the system. This requires being able to determine what the file is renamed to during the bulk import process. Given that information we would be able to scan the accumulo.metadata table to find its matching entry. We realize that there is a race condition here in which the GC could have removed it before verification could take place. That situation could be handled by looking in the GC logs which is not very clean but doable. We could of course monitor the master log to determine the file mapping as well but I was hoping for a cleaner solution.

One possibility is to actually include the name of the original file in the key or value within the file column family of the accumulo metadata. Another possibility is to have the master pass back the list of file name mappings to the client. The later could be achieved by writing a mapping file into the directory that was being imported or alternatively the failure directory.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the bulk import process and how it records entries in the accumulo.metadata table. Compare the proposed original-file mapping, client-returned mappings, and mapping-file approaches, including the stated GC and master-log race concerns. Done means selecting and implementing a defined way to verify the imported file mapping, with validation of the failure and GC cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.