Provide mechanism to make more fine grained caching decisions.
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
Some ranges or locality groups within a table or tablet may never benefit from being cached and caching them will unnecessarily pollute the cache. Currently caching is only tune-able at a table and scan level, but tuning based on data within a table is not possible.
**Describe the solution you'd like**
A pluggable mechanism to tune what specific data in a table should and should not be cached.
Prior to loading a block into cache the information in [IndexEntry](https://github.com/apache/accumulo/blob/77078f321fb492c0d80499973df04b386282e06f/core/src/main/java/org/apache/accumulo/core/file/rfile/MultiLevelIndex.java#L50) is known. The key field in IndexEntry is the last key in the datablock, it is easy in the RFIle code to obtain the previous index entry which has the last key for the previous block. Can use these two keys to construct a range that the data block covers. Also the RFile code knows the locality group it is operating on.
With this information an SPI interface like the following could be constructed. With this interface a user could somehow provide a `Predicate` that can make fine grained decisions on what data to actually cache.
```java
/**
* Provides the characteristics of block of data that is under consideration for caching.
*/
interface CacheRequest {
/**
* @return the number of entries in the cached block
*/
int getNumEntries();
/**
* @return the uncompressed size of the cached block
*/
long getUncompressedSize();
/**
* @return the compressed size of the cached block
*/
long getCompressedSize();
/**
* @return the name of the locality group of the cache block
*/
String getLocalityGroup();
/**
* @return the data range of the cache block
*/
Range getDataRange();
}
```
Not sure where this should be added to the SPI. It could be added to BlockCacheManager, but that will make configuriing this per table really cumbersome as that is configured at the server level. It could be added to the existing ScanDispatch SPI which already allows making some per scan caching decisions in the SPI. Scan dispatching can be configured per table already. Or maybe it could be added somewhere else.
Contributor guide
Research direction
Start with core/src/main/java/org/apache/accumulo/core/file/rfile/MultiLevelIndex.java and the RFile caching path, then compare the existing BlockCacheManager and ScanDispatch SPIs. Done means a documented, pluggable caching decision mechanism can inspect the block range, locality group, and size characteristics while remaining configurable per table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100