Exception during scan due to cached block eviction (Possible recurrence of issue described in #774)
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Bug Description**
It appears that we've encountered the same behavior as observed in #774 while testing our rel/2.0.1 deployment in AWS. For a period of time, we observed that our table scans would sporadically fail with the error below (maybe once or twice per hour over a period of a couple days of continuous query testing)
```
[scan.LookupTask] WARN : exception while doing multi-scan
java.lang.IllegalStateException: Block was evicted
at org.apache.accumulo.core.file.blockfile.cache.lru.CachedBlock.recordSize(CachedBlock.java:159)
at org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCache.cacheBlock(LruBlockCache.java:238)
at org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCache.cacheBlock(LruBlockCache.java:263)
at org.apache.accumulo.core.file.blockfile.cache.lru.SynchronousLoadingBlockCache.getBlock(SynchronousLoadingBlockCache.java:133)
at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:409)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:808)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:585)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:628)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:591)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:817)
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:997)
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:895)
at org.apache.accumulo.core.iterators.system.LocalityGroupIterator.seek(LocalityGroupIterator.java:281)
at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1437)
at org.apache.accumulo.server.problems.ProblemReportingIterator.seek(ProblemReportingIterator.java:103)
at org.apache.accumulo.core.iterators.system.MultiIterator.seek(MultiIterator.java:104)
at org.apache.accumulo.core.iterators.system.StatsIterator.seek(StatsIterator.java:63)
at org.apache.accumulo.core.iterators.system.DeletingIterator.seek(DeletingIterator.java:73)
at org.apache.accumulo.core.iterators.ServerSkippingIterator.seek(ServerSkippingIterator.java:52)
at org.apache.accumulo.core.iterators.system.ColumnFamilySkippingIterator.seek(ColumnFamilySkippingIterator.java:127)
at org.apache.accumulo.core.iterators.SynchronizedServerFilter.seek(SynchronizedServerFilter.java:56)
at org.apache.accumulo.core.iterators.WrappingIterator.seek(WrappingIterator.java:99)
at org.apache.accumulo.core.iterators.user.VersioningIterator.seek(VersioningIterator.java:82)
at aquery.iterators.Query.get_pageid(Query.java:453)
at aquery.iterators.Query.next(Query.java:351)
at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:174)
at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:145)
at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:641)
at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:770)
at org.apache.accumulo.tserver.scan.LookupTask.run(LookupTask.java:116)
at org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:54)
at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
```
**Versions:**
- **Accumulo:** 2.0.1
- **OS:** CentOS 7.7.1908
- **Java:** OpenJDK 1.8.0_272
- **Hadoop:** 3.0.0-cdh6.3.1
- **ZK:** 3.4.14
**Cluster Environment:**
- 12-node m5.2xlarge ec2 cluster; 10 tservers; 3 ZKs; Hadoop HA namenodes
- LRU cache config:
```
default | tserver.cache.data.size ........................... | 10%
site | @override ...................................... | 2G
system | @override ...................................... | 4G
default | tserver.cache.index.size .......................... | 25%
site | @override ...................................... | 2G
system | @override ...................................... | 1G
default | tserver.cache.manager.class ....................... | org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCacheManager
```
**Other relevant facts:**
- As best we can tell, the errors began to surface several hours after having switched `tserver.cache.manager.class` (via accumulo shell) from *[accumulo.ohc.OhcCacheManager](https://github.com/keith-turner/accumulo-ohc)* back to *LruBlockCacheManager*
- No intervening restart of Accumulo was performed after said config change in the shell
- At the time of the errors, the cluster was under medium-heavy load from concurrent user queries/scans and live ingest
- After a recent full restart of Accumulo, the errors seem to have ceased
There is no evidence in logs of this error having occurred prior to this instance (going back over a month), and several hours have lapsed since the full restart of Accumulo. No recurrence so far
Contributor guide
Research direction
Start with the cached-block classes in the stack trace, especially CachedBlock.java, LruBlockCache.java, SynchronousLoadingBlockCache.java, and CachableBlockFile.java, and compare the behavior with issue #774. Reproduce the scan failure around switching to LruBlockCacheManager without a restart, using the listed cache settings and load conditions. Done means the cause is identified and scans no longer fail with Block was evicted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100