apache / apache/accumulo

Re-evaluate metadata tables locality groups

Open
#2,161 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

While running a test of the new external compaction feature I Was looking at the metadata table config. External compactions add a new column family to the metadata table called ecomp. I noticed this new column family was not in the metadata table locality group config. Not sure if it should be. I think these configurations are pretty old and there may be other new column families. Would be good to reassess the locality group config against current column families and metadata table query patterns.

```
root@muchos> config -t accumulo.metadata -f group
-----------+-------------------------+------------------------------------------
SCOPE | NAME | VALUE
-----------+-------------------------+------------------------------------------
table | table.group.server .... | file,log,srv,future
table | table.group.tablet .... | ~tab,loc
default | table.groups.enabled .. |
table | @override .......... | tablet,server
-----------+-------------------------+------------------------------------------
```

```
$ accumulo rfile-info 'hdfs://accucluster/accumulo/tables/!0/table_info/A0002uqa.rf'
Reading file: hdfs://accucluster/accumulo/tables/!0/table_info/A0002uqa.rf
No on disk encryption detected.
RFile Version : 8

Locality group : tablet
Num blocks : 2
Index level 0 : 59 bytes 1 blocks
First key : +rep< ~tab:~pr [] 0 false
Last key : 2< ~tab:~pr [] 52040 false
Num entries : 1,411
Column families : [~tab, loc]
Locality group : server
Num blocks : 10
Index level 0 : 406 bytes 1 blocks
First key : +rep< srv:dir [] 0 false
Last key : 2< srv:time [] 135179 false
Num entries : 7,000
Column families : [log, file, srv, future]
Locality group :
Num blocks : 3
Index level 0 : 116 bytes 1 blocks
First key : 1< last:100000f2b52002c [] 132069 false
Last key : 2< last:100000f2b52001e [] 135179 false
Num entries : 793
Column families : [ecomp, last]

Meta block : BCFile.index
Raw size : 4 bytes
Compressed size : 12 bytes
Compression type : gz

Meta block : RFile.index
Raw size : 884 bytes
Compressed size : 559 bytes
Compression type : gz
```

Contributor guide

Open the contributing guide

Research direction

Start with the metadata table locality-group configuration shown by `config -t accumulo.metadata -f group`, then compare it with the current column families and metadata-table query patterns. Use the `rfile-info` output as a reference, including the default group containing `ecomp` and `last`; done means the configuration has been reassessed and any justified updates are documented or applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.