apache / apache/accumulo

Compaction property redesign

Open
#4,273 4 comments 0 reactions 1 assignee Claimed by @ddanielr View on GitHub
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

Had a discussion with @keith-turner and @ddanielr about some of the compaction configuration and SPI design, and we came up with some of these ideas that could be implemented:

One major unknown is what to call it. We went back and forth a bit on whether it should be `CompactionService*` or `CompactionPlanner*`

1. Remove user-facing SPI for CompactionPlanner and related config
2. Replace with CompactionServiceFactory:
a. `compaction.service.factory` holds the class name (singleton ref in ServerContext?)
b. `compaction.service.factory.config` holds a single string of all the factory's config
3. Suggested API (names can change later, this is just an idea)
```java
interface CompactionServiceFactory {
void init(PluginEnvironment env);
CompactionService forName(String service);
}

// configuration for a bounded named group / queue
class GroupConfig {
String name;
int maxQueueSize;
}

interface CompactionService {
Set getGroups();
List plan(tabletPlanningInformation);
}
```
4. `init` can validate the config, or re-validate periodically (implementation dependent)

Our default configuration for the `compaction.service.factory.config` could be (not pretty-printed, but can be copied/pasted into the property description in an HTML-friendly pretty-printed way):
```json
[
{
"meta": {
"maxOpenFilesPerJob": "30",
"groups": [
{
"name": "accumulo_meta_small",
"maxSize": "128M",
"maxJobs": "1000"
},
{
"name": "accumulo_meta_large",
"maxJobs": "1000"
}
]
},
"default": {
"maxOpenFilesPerJob": "30",
"groups": [
{
"name": "user_small",
"maxSize": "128M",
"maxJobs": "1000"
},
{
"name": "user_large",
"maxJobs": "1000"
}
]
}
}
]
```

One of the main benefits of this is having simplified configuration for user's configuration file. Also, we'll be able to set a default value in the DefaultConfiguration that actually lives in one place, and is easy for users to view for reference that represents the actual default behavior and override if they want to.

Re: #3981, #4034 , #4061

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.