Consider adding integrity checks into the metadata table's tablet schema
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
When reading the metadata table, a variety of problems could hypothetically occur. In order to try to prevent the worst of these from going undetected, we should consider adding some integrity checks into the metadata schema for tablets.
**Describe the solution you'd like**
A hash of each column family could work, so when you read a column family, such as those containing tablet files, you are guaranteed to be able to detect that you read every file entry, instead of just seeing a few of the many files in the tablet. You would have to read the hash entry for a column family in addition to all entries in that column family in order to verify the hash.
**Describe alternatives you've considered**
A hash of the metadata for an entire tablet could work, but it forces a full read of all the tablet's metadata entries, instead of allowing the scanner to just grab the columns it needs.
Contributor guide
Research direction
Start by locating the metadata table's tablet schema and the code that reads tablet column families. Investigate how a per-column-family hash could detect missing file entries without requiring a full metadata read; done would mean an agreed and implemented integrity check with validation of the relevant read paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100