apache / apache/accumulo

Consider adding integrity checks into the metadata table's tablet schema

Open
#3,699 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
When reading the metadata table, a variety of problems could hypothetically occur. In order to try to prevent the worst of these from going undetected, we should consider adding some integrity checks into the metadata schema for tablets.

**Describe the solution you'd like**
A hash of each column family could work, so when you read a column family, such as those containing tablet files, you are guaranteed to be able to detect that you read every file entry, instead of just seeing a few of the many files in the tablet. You would have to read the hash entry for a column family in addition to all entries in that column family in order to verify the hash.

**Describe alternatives you've considered**
A hash of the metadata for an entire tablet could work, but it forces a full read of all the tablet's metadata entries, instead of allowing the scanner to just grab the columns it needs.

Contributor guide

Open the contributing guide

Research direction

Start by locating the metadata table's tablet schema and the code that reads tablet column families. Investigate how a per-column-family hash could detect missing file entries without requiring a full metadata read; done would mean an agreed and implemented integrity check with validation of the relevant read paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.