apache / apache/accumulo

Avoid exceeding a user's configured max entries per DFS directory

Open
#6,413 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

In DFS, we use a structure like:
```
/tables///.rf
```

Some DFS implementations have limits on the number of entries per directory, and this may be tunable. Some users may tune this to be a small value, relative to the number of tablets on their system. The number of files in a tablet is tunable by users using compactions. But the number of tables and the number of tablets per table are areas that could pose a problem when considering the number of entries in a directory. The number of tablets is particularly important, since it's generally advantageous to have many tablets, in order to distribute work and avoid hotspots.

I propose a directory structure more like:
```
/tables/////.rf
```

For example, a `srv:dir` for a tablet in table `1a` in namespace `a` that used to look like `t-1234567`:

```
hdfs://server:port/accumulo/tables/1a/t-1234567/C0000000a.rf
```

instead uses `srv:dir` that looks like `t-12/34567` or `t-123/4567` or even `t-123/456/789` for longer tablet IDs, if needed, resulting in an absolute path that looks like:

```
hdfs://server:port/accumulo/tables/a/1a/t-12/34567/C0000000a.rf
```

The exact structure would need to be carefully considered during implementation to ensure that the hierarchy is sufficiently deep so that a reasonably configured max entry limit in DFS will not be at risk of being exceeded by Accumulo, even when users create many splits.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Accumulo constructs the DFS srv:dir path for tables, tablets, and files. Compare the proposed namespace and tablet-ID prefix/suffix layouts, then define a hierarchy deep enough that many tables, tablets, and splits do not exceed a configured per-directory entry limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.